[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:45:04 dist-test-slave-8tdl krb5kdc[1872](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:45:04 dist-test-slave-8tdl krb5kdc[1872](info): set up 2 sockets
Aug 06 02:45:04 dist-test-slave-8tdl krb5kdc[1872](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260806 02:45:05.346752 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.001s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:45:05 dist-test-slave-8tdl krb5kdc[1872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:45:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
I20260806 02:45:05.543655 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:33169
--webserver_interface=127.1.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:44237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:33169
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:06.000702 1885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:06.001099 1885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:06.001303 1885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:06.011250 1885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:06.011480 1885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:06.011581 1885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:06.011672 1885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:06.011750 1885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:45:06.024008 1885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:44237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:33169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:33169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:06.025485 1885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:06.027112 1885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:06.047667 1890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:06.047525 1891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:06.048204 1893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:06.049423 1885 server_base.cc:1034] running on GCE node
I20260806 02:45:06.050833 1885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:06.052860 1885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:06.054179 1885 hybrid_clock.cc:648] HybridClock initialized: now 1785984306054152 us; error 99 us; skew 500 ppm
I20260806 02:45:06.054780 1885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:45:06 dist-test-slave-8tdl krb5kdc[1872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984306, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:06.059051 1885 init.cc:377] Logged in from keytab as kudu/127.1.210.126@KRBTEST.COM (short username kudu)
I20260806 02:45:06.061132 1885 webserver.cc:460] Webserver started at http://127.1.210.126:38015/ using document root <none> and password file <none>
I20260806 02:45:06.062059 1885 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:06.062309 1885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:06.062774 1885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:06.067095 1885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "0e3eb2a0b10d4c35bba5c60cb10f05f9"
format_stamp: "Formatted at 2026-08-06 02:45:06 on dist-test-slave-8tdl"
server_key: "5a291b1ebfbbf6c2603895e243639cb2"
server_key_iv: "7e79a3b4d1042fb42265b06a34e31de3"
server_key_version: "encryptionkey@0"
I20260806 02:45:06.067939 1885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "0e3eb2a0b10d4c35bba5c60cb10f05f9"
format_stamp: "Formatted at 2026-08-06 02:45:06 on dist-test-slave-8tdl"
server_key: "5a291b1ebfbbf6c2603895e243639cb2"
server_key_iv: "7e79a3b4d1042fb42265b06a34e31de3"
server_key_version: "encryptionkey@0"
I20260806 02:45:06.075414 1885 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20260806 02:45:06.080235 1900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:06.081815 1885 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260806 02:45:06.082048 1885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "0e3eb2a0b10d4c35bba5c60cb10f05f9"
format_stamp: "Formatted at 2026-08-06 02:45:06 on dist-test-slave-8tdl"
server_key: "5a291b1ebfbbf6c2603895e243639cb2"
server_key_iv: "7e79a3b4d1042fb42265b06a34e31de3"
server_key_version: "encryptionkey@0"
I20260806 02:45:06.082329 1885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:06.105840 1885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:06.111243 1885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:06.111594 1885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:06.150882 1885 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:33169
I20260806 02:45:06.151062 1954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:33169 every 8 connection(s)
I20260806 02:45:06.152871 1885 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:45:06.158591 1955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:06.160672 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 1885
I20260806 02:45:06.161019 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:45:06.163480 1865 external_mini_cluster.cc:1428] Setting key 700331349591dce84a12bfc86949b698
I20260806 02:45:06.174258 1955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e3eb2a0b10d4c35bba5c60cb10f05f9: Bootstrap starting.
I20260806 02:45:06.178807 1955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e3eb2a0b10d4c35bba5c60cb10f05f9: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:06.180295 1955 log.cc:824] T 00000000000000000000000000000000 P 0e3eb2a0b10d4c35bba5c60cb10f05f9: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:06.184187 1955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e3eb2a0b10d4c35bba5c60cb10f05f9: No bootstrap required, opened a new log
I20260806 02:45:06.193606 1955 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e3eb2a0b10d4c35bba5c60cb10f05f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e3eb2a0b10d4c35bba5c60cb10f05f9" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 33169 } }
I20260806 02:45:06.194175 1955 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e3eb2a0b10d4c35bba5c60cb10f05f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:06.194370 1955 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e3eb2a0b10d4c35bba5c60cb10f05f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e3eb2a0b10d4c35bba5c60cb10f05f9, State: Initialized, Role: FOLLOWER
I20260806 02:45:06.195283 1955 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e3eb2a0b10d4c35bba5c60cb10f05f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e3eb2a0b10d4c35bba5c60cb10f05f9" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 33169 } }
I20260806 02:45:06.195636 1955 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e3eb2a0b10d4c35bba5c60cb10f05f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:45:06.195860 1955 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e3eb2a0b10d4c35bba5c60cb10f05f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:45:06.196061 1955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e3eb2a0b10d4c35bba5c60cb10f05f9 [term 0 FOLLOWER]: Advancing to term 1
Aug 06 02:45:06 dist-test-slave-8tdl krb5kdc[1872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:06.199653 1955 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e3eb2a0b10d4c35bba5c60cb10f05f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e3eb2a0b10d4c35bba5c60cb10f05f9" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 33169 } }
I20260806 02:45:06.200634 1955 leader_election.cc:302] T 00000000000000000000000000000000 P 0e3eb2a0b10d4c35bba5c60cb10f05f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e3eb2a0b10d4c35bba5c60cb10f05f9; no voters:
I20260806 02:45:06.202517 1955 leader_election.cc:288] T 00000000000000000000000000000000 P 0e3eb2a0b10d4c35bba5c60cb10f05f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:45:06.202837 1960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e3eb2a0b10d4c35bba5c60cb10f05f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:06.206905 1955 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e3eb2a0b10d4c35bba5c60cb10f05f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:06.207592 1960 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e3eb2a0b10d4c35bba5c60cb10f05f9 [term 1 LEADER]: Becoming Leader. State: Replica: 0e3eb2a0b10d4c35bba5c60cb10f05f9, State: Running, Role: LEADER
I20260806 02:45:06.208345 1960 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e3eb2a0b10d4c35bba5c60cb10f05f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e3eb2a0b10d4c35bba5c60cb10f05f9" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 33169 } }
I20260806 02:45:06.212456 1962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e3eb2a0b10d4c35bba5c60cb10f05f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e3eb2a0b10d4c35bba5c60cb10f05f9. Latest consensus state: current_term: 1 leader_uuid: "0e3eb2a0b10d4c35bba5c60cb10f05f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e3eb2a0b10d4c35bba5c60cb10f05f9" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 33169 } } }
I20260806 02:45:06.213029 1962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e3eb2a0b10d4c35bba5c60cb10f05f9 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:06.213807 1961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e3eb2a0b10d4c35bba5c60cb10f05f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e3eb2a0b10d4c35bba5c60cb10f05f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e3eb2a0b10d4c35bba5c60cb10f05f9" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 33169 } } }
I20260806 02:45:06.214263 1961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e3eb2a0b10d4c35bba5c60cb10f05f9 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:06.221992 1957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:06.168081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57466 (local address 127.1.210.126:33169)
0806 02:45:06.169727 (+ 1646us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:06.169746 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:06.171006 (+ 1260us) server_negotiation.cc:407] Connection header received
0806 02:45:06.182857 (+ 11851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:06.182935 (+ 78us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:06.183432 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:06.183913 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:06.187624 (+ 3711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:06.189487 (+ 1863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:06.191892 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:06.192765 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:06.193101 (+ 336us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:06.202367 (+ 9266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:06.202382 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:06.202404 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:06.202461 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:06.208179 (+ 5718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:06.209781 (+ 1602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:06.209803 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:06.209814 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:06.210203 (+ 389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:06.211450 (+ 1247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:06.211455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:06.211459 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:06.212708 (+ 1249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:06.213338 (+ 630us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:06.218482 (+ 5144us) server_negotiation.cc:299] Negotiation successful
0806 02:45:06.219285 (+ 803us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1346,"thread_start_us":981,"threads_started":1}
I20260806 02:45:06.234192 1968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:06.245644 1968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:45:06.261981 1968 catalog_manager.cc:1347] Generated new cluster ID: cd240208dfc84fbb98e68836ef87107a
I20260806 02:45:06.262202 1968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:06.274255 1968 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:45:06.276715 1968 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:06.291536 1968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e3eb2a0b10d4c35bba5c60cb10f05f9: Generated new TSK 0
I20260806 02:45:06.293205 1968 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab.
Entry for principal kudu/127.1.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab.
I20260806 02:45:06.359300 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.65:0
--local_ip_for_outbound_sockets=127.1.210.65
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:33169
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:44237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:06.890738 1982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:06.891208 1982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:06.891443 1982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:06.900846 1982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:06.901141 1982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:06.901295 1982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.65
I20260806 02:45:06.913213 1982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:44237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:33169
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.65
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:06.914721 1982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:06.916227 1982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:06.931905 1990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:06.932107 1987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:06.932124 1988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:06.932997 1982 server_base.cc:1034] running on GCE node
I20260806 02:45:06.933734 1982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:06.935002 1982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:06.936323 1982 hybrid_clock.cc:648] HybridClock initialized: now 1785984306936166 us; error 62 us; skew 500 ppm
I20260806 02:45:06.936813 1982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:45:06 dist-test-slave-8tdl krb5kdc[1872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984306, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:06.942101 1982 init.cc:377] Logged in from keytab as kudu/127.1.210.65@KRBTEST.COM (short username kudu)
I20260806 02:45:06.943799 1982 webserver.cc:460] Webserver started at http://127.1.210.65:41683/ using document root <none> and password file <none>
I20260806 02:45:06.944597 1982 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:06.944780 1982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:06.945190 1982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:06.948652 1982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-0/data/instance:
uuid: "c52a88b70d394c669692961ea965279b"
format_stamp: "Formatted at 2026-08-06 02:45:06 on dist-test-slave-8tdl"
server_key: "6807b97283a30d79aa43df95fd2f890a"
server_key_iv: "f7ce4622a789a270c7f5fee5b2d1c897"
server_key_version: "encryptionkey@0"
I20260806 02:45:06.949549 1982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance:
uuid: "c52a88b70d394c669692961ea965279b"
format_stamp: "Formatted at 2026-08-06 02:45:06 on dist-test-slave-8tdl"
server_key: "6807b97283a30d79aa43df95fd2f890a"
server_key_iv: "f7ce4622a789a270c7f5fee5b2d1c897"
server_key_version: "encryptionkey@0"
I20260806 02:45:06.956207 1982 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260806 02:45:06.960445 1997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:06.962235 1982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:45:06.962455 1982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-0/wal
uuid: "c52a88b70d394c669692961ea965279b"
format_stamp: "Formatted at 2026-08-06 02:45:06 on dist-test-slave-8tdl"
server_key: "6807b97283a30d79aa43df95fd2f890a"
server_key_iv: "f7ce4622a789a270c7f5fee5b2d1c897"
server_key_version: "encryptionkey@0"
I20260806 02:45:06.962715 1982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:06.989991 1982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:06.995385 1982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:06.995723 1982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:06.996786 1982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:06.998370 1982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:06.998478 1982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:06.998634 1982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:06.998739 1982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260806 02:45:07.060391 1982 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.65:43837
I20260806 02:45:07.060563 2112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.65:43837 every 8 connection(s)
I20260806 02:45:07.062186 1982 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
I20260806 02:45:07.064851 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 1982
I20260806 02:45:07.065146 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance
I20260806 02:45:07.065860 1865 external_mini_cluster.cc:1428] Setting key 422d9358a98927538069f5bfd705a320
Aug 06 02:45:07 dist-test-slave-8tdl krb5kdc[1872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984306, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.210.66@KRBTEST.COM; defaulting to no policy
I20260806 02:45:07.087039 2115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:07.065810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:56167 (local address 127.1.210.126:33169)
0806 02:45:07.066629 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:07.066637 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:07.068159 (+ 1522us) server_negotiation.cc:407] Connection header received
0806 02:45:07.070618 (+ 2459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:07.070624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:07.070728 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:07.070828 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:07.072732 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:07.073454 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:07.075244 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:07.075650 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:07.075785 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:07.080189 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:07.080195 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:07.080199 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:07.080236 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:07.083482 (+ 3246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:07.084310 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:07.084316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:07.084326 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:07.084496 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:07.084968 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:07.084978 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:07.084980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:07.085297 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:07.085463 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:07.086223 (+ 760us) server_negotiation.cc:299] Negotiation successful
0806 02:45:07.086464 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":498,"threads_started":1}
I20260806 02:45:07.088250 2116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:07.066614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33169 (local address 127.1.210.65:56167)
0806 02:45:07.068004 (+ 1390us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:07.068049 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:07.070324 (+ 2275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:07.071301 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:07.071310 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:07.071849 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:07.072497 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:07.072515 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:07.073600 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:07.073608 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:07.075101 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:07.075114 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:07.075963 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:07.075974 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:07.076241 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:07.077016 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:07.077039 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:07.079912 (+ 2873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:07.083663 (+ 3751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:07.083673 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:07.083688 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:07.084171 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:07.084617 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:07.084622 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:07.084625 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:07.084834 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:07.085437 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:07.085446 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:07.085828 (+ 382us) client_negotiation.cc:769] Sending connection context
0806 02:45:07.086127 (+ 299us) client_negotiation.cc:240] Negotiation successful
0806 02:45:07.086773 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1031,"thread_start_us":692,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.66@KRBTEST.COM" created.
I20260806 02:45:07.090180 2113 heartbeater.cc:347] Connected to a master server at 127.1.210.126:33169
I20260806 02:45:07.090550 2113 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:07.091429 2113 heartbeater.cc:511] Master 127.1.210.126:33169 requested a full tablet report, sending...
I20260806 02:45:07.094082 1918 ts_manager.cc:194] Registered new tserver with Master: c52a88b70d394c669692961ea965279b (127.1.210.65:43837)
I20260806 02:45:07.096067 1918 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.65@KRBTEST.COM'} at 127.1.210.65:56167
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab.
Entry for principal kudu/127.1.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab.
I20260806 02:45:07.115494 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.66:0
--local_ip_for_outbound_sockets=127.1.210.66
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:33169
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:44237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:07.602876 2119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:07.603193 2119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:07.603363 2119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:07.612926 2119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:07.613145 2119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:07.613411 2119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.66
I20260806 02:45:07.626341 2119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:44237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:33169
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.66
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:07.627763 2119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:07.629419 2119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:07.646248 2124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:07.646754 2127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:07.646773 2125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:07.648494 2119 server_base.cc:1034] running on GCE node
I20260806 02:45:07.649168 2119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:07.650194 2119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:07.651451 2119 hybrid_clock.cc:648] HybridClock initialized: now 1785984307651424 us; error 92 us; skew 500 ppm
I20260806 02:45:07.651899 2119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:45:07 dist-test-slave-8tdl krb5kdc[1872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984307, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:07.656502 2119 init.cc:377] Logged in from keytab as kudu/127.1.210.66@KRBTEST.COM (short username kudu)
I20260806 02:45:07.658911 2119 webserver.cc:460] Webserver started at http://127.1.210.66:41419/ using document root <none> and password file <none>
I20260806 02:45:07.659852 2119 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:07.660096 2119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:07.660583 2119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:07.664664 2119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-1/data/instance:
uuid: "4dfea13d27d14abda4d1af495814baa0"
format_stamp: "Formatted at 2026-08-06 02:45:07 on dist-test-slave-8tdl"
server_key: "c14f1ea41010084004391fb9b7ac9007"
server_key_iv: "e4c49fda603f64f46eae98d5cddd4ece"
server_key_version: "encryptionkey@0"
I20260806 02:45:07.665726 2119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance:
uuid: "4dfea13d27d14abda4d1af495814baa0"
format_stamp: "Formatted at 2026-08-06 02:45:07 on dist-test-slave-8tdl"
server_key: "c14f1ea41010084004391fb9b7ac9007"
server_key_iv: "e4c49fda603f64f46eae98d5cddd4ece"
server_key_version: "encryptionkey@0"
I20260806 02:45:07.673365 2119 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260806 02:45:07.679039 2134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:07.680784 2119 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:45:07.681087 2119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-1/wal
uuid: "4dfea13d27d14abda4d1af495814baa0"
format_stamp: "Formatted at 2026-08-06 02:45:07 on dist-test-slave-8tdl"
server_key: "c14f1ea41010084004391fb9b7ac9007"
server_key_iv: "e4c49fda603f64f46eae98d5cddd4ece"
server_key_version: "encryptionkey@0"
I20260806 02:45:07.681388 2119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:07.701514 2119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:07.707839 2119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:07.708379 2119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:07.709945 2119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:07.711838 2119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:07.711971 2119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:07.712074 2119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:07.712127 2119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:07.779523 2119 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.66:43361
I20260806 02:45:07.779677 2246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.66:43361 every 8 connection(s)
I20260806 02:45:07.781000 2119 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
I20260806 02:45:07.790555 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 2119
I20260806 02:45:07.790812 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance
I20260806 02:45:07.791277 1865 external_mini_cluster.cc:1428] Setting key eb65348e3a3a226a2e1335939d86ba2d
Aug 06 02:45:07 dist-test-slave-8tdl krb5kdc[1872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984307, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:07.807271 2249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:07.783919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:51461 (local address 127.1.210.126:33169)
0806 02:45:07.784509 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:07.784513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:07.785353 (+ 840us) server_negotiation.cc:407] Connection header received
0806 02:45:07.787605 (+ 2252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:07.787612 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:07.787713 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:07.787857 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:07.789485 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:07.790411 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:07.792777 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:07.793512 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:07.793613 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:07.799371 (+ 5758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:07.799376 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:07.799380 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:07.799418 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:07.803086 (+ 3668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:07.803988 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:07.803998 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:07.804004 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:07.804200 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:07.804744 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:07.804753 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:07.804760 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:07.805315 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:07.805507 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:07.806302 (+ 795us) server_negotiation.cc:299] Negotiation successful
0806 02:45:07.806620 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":354,"threads_started":1}
I20260806 02:45:07.808070 2250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:07.784344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33169 (local address 127.1.210.66:51461)
0806 02:45:07.785178 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:07.785223 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:07.787256 (+ 2033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:07.788125 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:07.788134 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:07.788691 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:07.789286 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:07.789304 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:07.790769 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:07.790775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:07.792634 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:07.792648 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:07.793766 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:07.793774 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:07.794164 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:07.795117 (+ 953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:07.795174 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:07.799075 (+ 3901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:07.803302 (+ 4227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:07.803310 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:07.803326 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:07.803807 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:07.804359 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:07.804363 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:07.804365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:07.804596 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:07.805469 (+ 873us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:07.805478 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:07.805857 (+ 379us) client_negotiation.cc:769] Sending connection context
0806 02:45:07.806129 (+ 272us) client_negotiation.cc:240] Negotiation successful
0806 02:45:07.806640 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":351,"threads_started":1}
I20260806 02:45:07.810267 2247 heartbeater.cc:347] Connected to a master server at 127.1.210.126:33169
I20260806 02:45:07.810606 2247 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:07.811750 2247 heartbeater.cc:511] Master 127.1.210.126:33169 requested a full tablet report, sending...
I20260806 02:45:07.813727 1918 ts_manager.cc:194] Registered new tserver with Master: 4dfea13d27d14abda4d1af495814baa0 (127.1.210.66:43361)
I20260806 02:45:07.814633 1918 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.66@KRBTEST.COM'} at 127.1.210.66:51461
WARNING: no policy specified for kudu/127.1.210.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab.
Entry for principal kudu/127.1.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab.
I20260806 02:45:07.847879 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.67:0
--local_ip_for_outbound_sockets=127.1.210.67
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:33169
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:44237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:45:08.099717 2113 heartbeater.cc:503] Master 127.1.210.126:33169 was elected leader, sending a full tablet report...
W20260806 02:45:08.344437 2253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:08.344888 2253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:08.345199 2253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:08.353753 2253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:08.353924 2253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:08.354110 2253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.67
I20260806 02:45:08.366004 2253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:44237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:33169
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.67
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:08.367539 2253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:08.369051 2253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:08.384697 2259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:08.384851 2262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:08.385115 2260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:08.386631 2253 server_base.cc:1034] running on GCE node
I20260806 02:45:08.387440 2253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:08.388866 2253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:08.390240 2253 hybrid_clock.cc:648] HybridClock initialized: now 1785984308390138 us; error 45 us; skew 500 ppm
I20260806 02:45:08.390625 2253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:45:08 dist-test-slave-8tdl krb5kdc[1872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984308, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:08.397126 2253 init.cc:377] Logged in from keytab as kudu/127.1.210.67@KRBTEST.COM (short username kudu)
I20260806 02:45:08.399340 2253 webserver.cc:460] Webserver started at http://127.1.210.67:34455/ using document root <none> and password file <none>
I20260806 02:45:08.400265 2253 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:08.400506 2253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:08.400966 2253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:08.404872 2253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-2/data/instance:
uuid: "c0c895b25d254c1591c5ba813c8eeb47"
format_stamp: "Formatted at 2026-08-06 02:45:08 on dist-test-slave-8tdl"
server_key: "ace5f39408ec2e26e541b0f72d92f2b3"
server_key_iv: "de616a3adf3001e62d666c0317eae7bd"
server_key_version: "encryptionkey@0"
I20260806 02:45:08.405767 2253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance:
uuid: "c0c895b25d254c1591c5ba813c8eeb47"
format_stamp: "Formatted at 2026-08-06 02:45:08 on dist-test-slave-8tdl"
server_key: "ace5f39408ec2e26e541b0f72d92f2b3"
server_key_iv: "de616a3adf3001e62d666c0317eae7bd"
server_key_version: "encryptionkey@0"
I20260806 02:45:08.412649 2253 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260806 02:45:08.417761 2269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:08.419468 2253 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260806 02:45:08.419859 2253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-2/wal
uuid: "c0c895b25d254c1591c5ba813c8eeb47"
format_stamp: "Formatted at 2026-08-06 02:45:08 on dist-test-slave-8tdl"
server_key: "ace5f39408ec2e26e541b0f72d92f2b3"
server_key_iv: "de616a3adf3001e62d666c0317eae7bd"
server_key_version: "encryptionkey@0"
I20260806 02:45:08.420199 2253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:08.447909 2253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:08.453689 2253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:08.454169 2253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:08.455996 2253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:08.457536 2253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:08.457616 2253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:08.457754 2253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:08.457806 2253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:08.532593 2384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.67:34751 every 8 connection(s)
I20260806 02:45:08.532675 2253 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.67:34751
I20260806 02:45:08.534111 2253 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
I20260806 02:45:08.542748 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 2253
I20260806 02:45:08.543078 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance
I20260806 02:45:08.543740 1865 external_mini_cluster.cc:1428] Setting key 86cfd9be22c6040ccf6b9add07b8d899
Aug 06 02:45:08 dist-test-slave-8tdl krb5kdc[1872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984308, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:08.558811 2387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.537313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:60707 (local address 127.1.210.126:33169)
0806 02:45:08.537933 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:08.537940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:08.539327 (+ 1387us) server_negotiation.cc:407] Connection header received
0806 02:45:08.542195 (+ 2868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:08.542204 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:08.542323 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:08.542531 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:08.544453 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.545546 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.547804 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.548360 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.548462 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:08.552532 (+ 4070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:08.552537 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:08.552541 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:08.552575 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:08.555486 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.556410 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.556414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.556416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.556542 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.557069 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.557072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.557074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.557339 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:08.557457 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:08.558238 (+ 781us) server_negotiation.cc:299] Negotiation successful
0806 02:45:08.558403 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":321,"threads_started":1}
I20260806 02:45:08.560149 2388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.538073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33169 (local address 127.1.210.67:60707)
0806 02:45:08.539091 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:08.539134 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:08.541838 (+ 2704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:08.542821 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:08.542831 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:08.543520 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:08.544241 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:08.544259 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.545742 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.545750 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:08.547649 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:08.547665 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.548668 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.548676 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:08.548904 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:08.549632 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:08.549655 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:08.552271 (+ 2616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:08.555632 (+ 3361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:08.555640 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:08.555654 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:08.556292 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:08.556695 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:08.556699 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:08.556702 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:08.556909 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:08.557507 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:08.557513 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:08.557868 (+ 355us) client_negotiation.cc:769] Sending connection context
0806 02:45:08.558138 (+ 270us) client_negotiation.cc:240] Negotiation successful
0806 02:45:08.558568 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":343,"threads_started":1}
I20260806 02:45:08.561983 2385 heartbeater.cc:347] Connected to a master server at 127.1.210.126:33169
I20260806 02:45:08.562356 2385 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:08.563287 2385 heartbeater.cc:511] Master 127.1.210.126:33169 requested a full tablet report, sending...
I20260806 02:45:08.565038 1917 ts_manager.cc:194] Registered new tserver with Master: c0c895b25d254c1591c5ba813c8eeb47 (127.1.210.67:34751)
I20260806 02:45:08.566571 1917 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.67@KRBTEST.COM'} at 127.1.210.67:60707
I20260806 02:45:08.573942 1865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:45:08 dist-test-slave-8tdl krb5kdc[1872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984308, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 02:45:08 dist-test-slave-8tdl krb5kdc[1872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984308, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.210.65@KRBTEST.COM
I20260806 02:45:08.636308 2393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.616493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38802 (local address 127.1.210.65:43837)
0806 02:45:08.617293 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:08.617301 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:08.617330 (+ 29us) server_negotiation.cc:407] Connection header received
0806 02:45:08.617442 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:08.617448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:08.617677 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:08.617814 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:08.619698 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.620806 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.622158 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.622965 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.623114 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:08.628721 (+ 5607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:08.628735 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:08.628752 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:08.628815 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:08.632713 (+ 3898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.633455 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.633462 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.633464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.633638 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.634094 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.634099 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.634103 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.634570 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:08.635059 (+ 489us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:08.635346 (+ 287us) server_negotiation.cc:299] Negotiation successful
0806 02:45:08.635726 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":432,"threads_started":1}
W20260806 02:45:08.637809 2025 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38802
Aug 06 02:45:08 dist-test-slave-8tdl krb5kdc[1872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 02:45:08 dist-test-slave-8tdl krb5kdc[1872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.65@KRBTEST.COM
I20260806 02:45:08.690445 2393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.672936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38812 (local address 127.1.210.65:43837)
0806 02:45:08.673206 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:08.673217 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:08.673763 (+ 546us) server_negotiation.cc:407] Connection header received
0806 02:45:08.674005 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:08.674011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:08.674146 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:08.674287 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:08.676235 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.676938 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.678579 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.679182 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.679291 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:08.684198 (+ 4907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:08.684204 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:08.684208 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:08.684249 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:08.687293 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.688206 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.688210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.688213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.688352 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.688855 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.688859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.688861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.689195 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:08.689401 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:08.689816 (+ 415us) server_negotiation.cc:299] Negotiation successful
0806 02:45:08.690016 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 02:45:08.692385 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 1982
I20260806 02:45:08.707989 2393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.693392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38826 (local address 127.1.210.65:43837)
0806 02:45:08.693626 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:08.693632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:08.694163 (+ 531us) server_negotiation.cc:407] Connection header received
0806 02:45:08.694365 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:08.694369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:08.694427 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:08.694538 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:08.696145 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.696844 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.698327 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.698827 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.698962 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:08.701726 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:08.701732 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:08.701736 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:08.701772 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:08.704773 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.705568 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.705577 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.705580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.705798 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.706269 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.706278 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.706282 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.706582 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:08.706934 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:08.707238 (+ 304us) server_negotiation.cc:299] Negotiation successful
0806 02:45:08.707504 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:45:08.708930 2108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:45:08.798492 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 1982
I20260806 02:45:08.815866 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 2119
I20260806 02:45:08.818089 2247 heartbeater.cc:503] Master 127.1.210.126:33169 was elected leader, sending a full tablet report...
Aug 06 02:45:08 dist-test-slave-8tdl krb5kdc[1872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.66@KRBTEST.COM
I20260806 02:45:08.835317 2400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.817079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36092 (local address 127.1.210.66:43361)
0806 02:45:08.818143 (+ 1064us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:08.818152 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:08.818202 (+ 50us) server_negotiation.cc:407] Connection header received
0806 02:45:08.818291 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:08.818303 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:08.818634 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:08.818833 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:08.820478 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.821698 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.823244 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.823941 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.824049 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:08.828896 (+ 4847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:08.828911 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:08.828927 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:08.828968 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:08.832293 (+ 3325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.833075 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.833083 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.833086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.833229 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.833699 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.833703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.833705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.834104 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:08.834367 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:08.834601 (+ 234us) server_negotiation.cc:299] Negotiation successful
0806 02:45:08.834898 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":809,"thread_start_us":417,"threads_started":1}
I20260806 02:45:08.836694 2242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:45:08.924149 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 2119
I20260806 02:45:08.942055 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 2253
Aug 06 02:45:08 dist-test-slave-8tdl krb5kdc[1872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.67@KRBTEST.COM
I20260806 02:45:08.960217 2402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:08.943189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37668 (local address 127.1.210.67:34751)
0806 02:45:08.943715 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:08.943722 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:08.943753 (+ 31us) server_negotiation.cc:407] Connection header received
0806 02:45:08.943813 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:08.943820 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:08.944054 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:08.944236 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:08.945823 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.947085 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.948460 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:08.949037 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:08.949152 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:08.953803 (+ 4651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:08.953817 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:08.953834 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:08.953882 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:08.957196 (+ 3314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.957958 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.957967 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.957969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.958110 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:08.958531 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:08.958534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:08.958536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:08.958972 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:08.959224 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:08.959545 (+ 321us) server_negotiation.cc:299] Negotiation successful
0806 02:45:08.959783 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":238,"threads_started":1}
I20260806 02:45:08.961550 2380 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:45:09.047329 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 2253
I20260806 02:45:09.064868 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 1885
I20260806 02:45:09.066174 1950 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:45:09.171468 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 1885
2026-08-06T02:45:09Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4970 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:45:09 dist-test-slave-8tdl krb5kdc[2407](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:45:09 dist-test-slave-8tdl krb5kdc[2407](info): set up 2 sockets
Aug 06 02:45:09 dist-test-slave-8tdl krb5kdc[2407](info): commencing operation
krb5kdc: starting...
W20260806 02:45:10.335393 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.006s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:45:10 dist-test-slave-8tdl krb5kdc[2407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:45:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
I20260806 02:45:10.540109 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:42041
--webserver_interface=127.1.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:36465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:42041
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:10.877468 2420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:10.877840 2420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:10.878003 2420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:10.885484 2420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:10.885610 2420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:10.885704 2420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:10.885813 2420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:10.885877 2420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:45:10.896521 2420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:36465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:42041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:42041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:10.898058 2420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:10.899580 2420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:10.912446 2425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:10.912356 2427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:10.912408 2429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:10.913340 2420 server_base.cc:1034] running on GCE node
I20260806 02:45:10.914249 2420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:10.915865 2420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:10.917126 2420 hybrid_clock.cc:648] HybridClock initialized: now 1785984310917057 us; error 67 us; skew 500 ppm
I20260806 02:45:10.917517 2420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:45:10 dist-test-slave-8tdl krb5kdc[2407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984310, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:10.924597 2420 init.cc:377] Logged in from keytab as kudu/127.1.210.126@KRBTEST.COM (short username kudu)
I20260806 02:45:10.926307 2420 webserver.cc:460] Webserver started at http://127.1.210.126:40187/ using document root <none> and password file <none>
I20260806 02:45:10.927170 2420 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:10.927325 2420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:10.927662 2420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:10.931025 2420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "88a0096f2e844760b0b040c3fbd2b40e"
format_stamp: "Formatted at 2026-08-06 02:45:10 on dist-test-slave-8tdl"
server_key: "e22405d697d3d7105bea24bc1fa37bba"
server_key_iv: "c173a0c478147b3b81026e4fad071dfa"
server_key_version: "encryptionkey@0"
I20260806 02:45:10.931885 2420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "88a0096f2e844760b0b040c3fbd2b40e"
format_stamp: "Formatted at 2026-08-06 02:45:10 on dist-test-slave-8tdl"
server_key: "e22405d697d3d7105bea24bc1fa37bba"
server_key_iv: "c173a0c478147b3b81026e4fad071dfa"
server_key_version: "encryptionkey@0"
I20260806 02:45:10.937644 2420 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260806 02:45:10.941605 2436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:10.943317 2420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:10.943540 2420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "88a0096f2e844760b0b040c3fbd2b40e"
format_stamp: "Formatted at 2026-08-06 02:45:10 on dist-test-slave-8tdl"
server_key: "e22405d697d3d7105bea24bc1fa37bba"
server_key_iv: "c173a0c478147b3b81026e4fad071dfa"
server_key_version: "encryptionkey@0"
I20260806 02:45:10.943768 2420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:10.965243 2420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:10.969811 2420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:10.970124 2420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:10.989715 2420 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:42041
I20260806 02:45:10.989833 2487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:42041 every 8 connection(s)
I20260806 02:45:10.991186 2420 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:45:10.995174 2488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:11.000403 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 2420
I20260806 02:45:11.000667 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:45:11.001484 1865 external_mini_cluster.cc:1428] Setting key c80e2ffcbdf9fd3a71c00e9635895190
I20260806 02:45:11.007620 2488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88a0096f2e844760b0b040c3fbd2b40e: Bootstrap starting.
I20260806 02:45:11.011080 2488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88a0096f2e844760b0b040c3fbd2b40e: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:11.012511 2488 log.cc:824] T 00000000000000000000000000000000 P 88a0096f2e844760b0b040c3fbd2b40e: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:11.016134 2488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88a0096f2e844760b0b040c3fbd2b40e: No bootstrap required, opened a new log
Aug 06 02:45:11 dist-test-slave-8tdl krb5kdc[2407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:11.024909 2488 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88a0096f2e844760b0b040c3fbd2b40e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88a0096f2e844760b0b040c3fbd2b40e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 42041 } }
I20260806 02:45:11.025368 2488 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88a0096f2e844760b0b040c3fbd2b40e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:11.025529 2488 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88a0096f2e844760b0b040c3fbd2b40e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a0096f2e844760b0b040c3fbd2b40e, State: Initialized, Role: FOLLOWER
I20260806 02:45:11.026155 2488 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88a0096f2e844760b0b040c3fbd2b40e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88a0096f2e844760b0b040c3fbd2b40e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 42041 } }
I20260806 02:45:11.026441 2488 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88a0096f2e844760b0b040c3fbd2b40e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:45:11.026604 2488 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88a0096f2e844760b0b040c3fbd2b40e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:45:11.026856 2488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88a0096f2e844760b0b040c3fbd2b40e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:11.027107 2492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:11.003812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42696 (local address 127.1.210.126:42041)
0806 02:45:11.004811 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:11.004827 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:11.004876 (+ 49us) server_negotiation.cc:407] Connection header received
0806 02:45:11.007037 (+ 2161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:11.007064 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:11.007543 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:11.008056 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:11.009760 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:11.011226 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:11.012572 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:11.013418 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:11.013676 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:11.018338 (+ 4662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:11.018349 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:11.018366 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:11.018414 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:11.022104 (+ 3690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:11.023038 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:11.023049 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:11.023056 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:11.023252 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:11.023699 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:11.023703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:11.023705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:11.024304 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:11.024619 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:11.025177 (+ 558us) server_negotiation.cc:299] Negotiation successful
0806 02:45:11.025721 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":734,"thread_start_us":499,"threads_started":1}
I20260806 02:45:11.029394 2488 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88a0096f2e844760b0b040c3fbd2b40e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88a0096f2e844760b0b040c3fbd2b40e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 42041 } }
I20260806 02:45:11.029937 2488 leader_election.cc:302] T 00000000000000000000000000000000 P 88a0096f2e844760b0b040c3fbd2b40e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88a0096f2e844760b0b040c3fbd2b40e; no voters:
I20260806 02:45:11.030540 2488 leader_election.cc:288] T 00000000000000000000000000000000 P 88a0096f2e844760b0b040c3fbd2b40e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:45:11.030707 2494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88a0096f2e844760b0b040c3fbd2b40e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:11.031628 2494 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88a0096f2e844760b0b040c3fbd2b40e [term 1 LEADER]: Becoming Leader. State: Replica: 88a0096f2e844760b0b040c3fbd2b40e, State: Running, Role: LEADER
I20260806 02:45:11.032227 2494 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88a0096f2e844760b0b040c3fbd2b40e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88a0096f2e844760b0b040c3fbd2b40e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 42041 } }
I20260806 02:45:11.032924 2488 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88a0096f2e844760b0b040c3fbd2b40e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:11.035902 2495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88a0096f2e844760b0b040c3fbd2b40e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88a0096f2e844760b0b040c3fbd2b40e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88a0096f2e844760b0b040c3fbd2b40e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 42041 } } }
I20260806 02:45:11.035899 2496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88a0096f2e844760b0b040c3fbd2b40e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88a0096f2e844760b0b040c3fbd2b40e. Latest consensus state: current_term: 1 leader_uuid: "88a0096f2e844760b0b040c3fbd2b40e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88a0096f2e844760b0b040c3fbd2b40e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 42041 } } }
I20260806 02:45:11.036396 2495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88a0096f2e844760b0b040c3fbd2b40e [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:11.036417 2496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88a0096f2e844760b0b040c3fbd2b40e [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:11.044952 2500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:11.052330 2500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:45:11.063570 2500 catalog_manager.cc:1347] Generated new cluster ID: c05bfd731f0f4830a5e557c2ecdd6f15
I20260806 02:45:11.063726 2500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:11.073321 2500 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:45:11.074819 2500 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:11.085665 2500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88a0096f2e844760b0b040c3fbd2b40e: Generated new TSK 0
I20260806 02:45:11.087006 2500 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab.
Entry for principal kudu/127.1.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab.
I20260806 02:45:11.139268 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.65:0
--local_ip_for_outbound_sockets=127.1.210.65
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:42041
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:36465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:11.474541 2515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:11.474896 2515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:11.475122 2515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:11.482473 2515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:11.482589 2515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:11.482805 2515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.65
I20260806 02:45:11.492807 2515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:36465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:42041
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.65
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:11.494316 2515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:11.495776 2515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:11.509151 2521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:11.509207 2523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:11.509227 2520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:11.515954 2515 server_base.cc:1034] running on GCE node
I20260806 02:45:11.516721 2515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:11.517807 2515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:11.519248 2515 hybrid_clock.cc:648] HybridClock initialized: now 1785984311519191 us; error 36 us; skew 500 ppm
I20260806 02:45:11.519616 2515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:45:11 dist-test-slave-8tdl krb5kdc[2407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984311, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:11.523159 2515 init.cc:377] Logged in from keytab as kudu/127.1.210.65@KRBTEST.COM (short username kudu)
I20260806 02:45:11.524830 2515 webserver.cc:460] Webserver started at http://127.1.210.65:34353/ using document root <none> and password file <none>
I20260806 02:45:11.525557 2515 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:11.525665 2515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:11.526072 2515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:11.529394 2515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-0/data/instance:
uuid: "474296ae66f84e53b4fb9582c4fd4493"
format_stamp: "Formatted at 2026-08-06 02:45:11 on dist-test-slave-8tdl"
server_key: "294ea75fa7dc46f4cc890c479c8eb6ea"
server_key_iv: "67df1c3faf014d2f34db966c7f9e56c0"
server_key_version: "encryptionkey@0"
I20260806 02:45:11.530192 2515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance:
uuid: "474296ae66f84e53b4fb9582c4fd4493"
format_stamp: "Formatted at 2026-08-06 02:45:11 on dist-test-slave-8tdl"
server_key: "294ea75fa7dc46f4cc890c479c8eb6ea"
server_key_iv: "67df1c3faf014d2f34db966c7f9e56c0"
server_key_version: "encryptionkey@0"
I20260806 02:45:11.536048 2515 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 02:45:11.540270 2530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:11.542196 2515 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:45:11.542505 2515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-0/wal
uuid: "474296ae66f84e53b4fb9582c4fd4493"
format_stamp: "Formatted at 2026-08-06 02:45:11 on dist-test-slave-8tdl"
server_key: "294ea75fa7dc46f4cc890c479c8eb6ea"
server_key_iv: "67df1c3faf014d2f34db966c7f9e56c0"
server_key_version: "encryptionkey@0"
I20260806 02:45:11.542778 2515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:11.578598 2515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:11.583266 2515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:11.583519 2515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:11.584561 2515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:11.586169 2515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:11.586242 2515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:11.586334 2515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:11.586380 2515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:11.621865 2515 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.65:33043
I20260806 02:45:11.621989 2645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.65:33043 every 8 connection(s)
I20260806 02:45:11.623296 2515 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
I20260806 02:45:11.629832 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 2515
I20260806 02:45:11.630088 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance
I20260806 02:45:11.630779 1865 external_mini_cluster.cc:1428] Setting key 03648d758df66cdee6a3266db6a49cc0
Aug 06 02:45:11 dist-test-slave-8tdl krb5kdc[2407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984311, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:11.646216 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:11.625977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:51021 (local address 127.1.210.126:42041)
0806 02:45:11.626398 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:11.626403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:11.627578 (+ 1175us) server_negotiation.cc:407] Connection header received
0806 02:45:11.629522 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:11.629528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:11.629628 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:11.629752 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:11.631510 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:11.632553 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:11.634294 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:11.634791 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:11.634977 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:11.638737 (+ 3760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:11.638744 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:11.638747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:11.638782 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:11.642334 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:11.643244 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:11.643253 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:11.643259 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:11.643419 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:11.643928 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:11.643937 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:11.643943 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:11.644351 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:11.644518 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:11.645396 (+ 878us) server_negotiation.cc:299] Negotiation successful
0806 02:45:11.645701 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":214,"threads_started":1}
I20260806 02:45:11.647257 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:11.626597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:42041 (local address 127.1.210.65:51021)
0806 02:45:11.627395 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:11.627442 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:11.629230 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:11.630055 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:11.630064 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:11.630637 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:11.631327 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:11.631345 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:11.632711 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:11.632717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:11.634155 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:11.634167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:11.635036 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:11.635046 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:11.635319 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:11.636121 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:11.636150 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:11.638441 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:11.642512 (+ 4071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:11.642520 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:11.642536 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:11.643107 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:11.643555 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:11.643558 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:11.643561 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:11.643760 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:11.644502 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:11.644509 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:11.644926 (+ 417us) client_negotiation.cc:769] Sending connection context
0806 02:45:11.645205 (+ 279us) client_negotiation.cc:240] Negotiation successful
0806 02:45:11.645653 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":329,"threads_started":1}
I20260806 02:45:11.649012 2646 heartbeater.cc:347] Connected to a master server at 127.1.210.126:42041
I20260806 02:45:11.649343 2646 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:11.650098 2646 heartbeater.cc:511] Master 127.1.210.126:42041 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.210.66@KRBTEST.COM; defaulting to no policy
I20260806 02:45:11.652735 2453 ts_manager.cc:194] Registered new tserver with Master: 474296ae66f84e53b4fb9582c4fd4493 (127.1.210.65:33043)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.66@KRBTEST.COM" created.
I20260806 02:45:11.655001 2453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.65@KRBTEST.COM'} at 127.1.210.65:51021
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab.
Entry for principal kudu/127.1.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab.
I20260806 02:45:11.678736 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.66:0
--local_ip_for_outbound_sockets=127.1.210.66
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:42041
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:36465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:12.018821 2652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:12.019173 2652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:12.019351 2652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:12.027137 2652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:12.027292 2652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:12.027468 2652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.66
I20260806 02:45:12.037827 2652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:36465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:42041
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.66
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:12.039342 2652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:12.040818 2652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:12.055460 2660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:12.055622 2658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:12.056165 2657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:12.056725 2652 server_base.cc:1034] running on GCE node
I20260806 02:45:12.057408 2652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:12.058540 2652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:12.059787 2652 hybrid_clock.cc:648] HybridClock initialized: now 1785984312059742 us; error 48 us; skew 500 ppm
I20260806 02:45:12.060199 2652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:45:12 dist-test-slave-8tdl krb5kdc[2407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984312, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:12.063604 2652 init.cc:377] Logged in from keytab as kudu/127.1.210.66@KRBTEST.COM (short username kudu)
I20260806 02:45:12.065304 2652 webserver.cc:460] Webserver started at http://127.1.210.66:33763/ using document root <none> and password file <none>
I20260806 02:45:12.066012 2652 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:12.066116 2652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:12.066380 2652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:12.069638 2652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-1/data/instance:
uuid: "3bf1aa175aa149e6823c3f56d5748665"
format_stamp: "Formatted at 2026-08-06 02:45:12 on dist-test-slave-8tdl"
server_key: "ddc9f4a86c1e8c42c5d1d85c0f288cde"
server_key_iv: "6b9c2f734e33d2fa4cf7d7531fab1087"
server_key_version: "encryptionkey@0"
I20260806 02:45:12.070418 2652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance:
uuid: "3bf1aa175aa149e6823c3f56d5748665"
format_stamp: "Formatted at 2026-08-06 02:45:12 on dist-test-slave-8tdl"
server_key: "ddc9f4a86c1e8c42c5d1d85c0f288cde"
server_key_iv: "6b9c2f734e33d2fa4cf7d7531fab1087"
server_key_version: "encryptionkey@0"
I20260806 02:45:12.075994 2652 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 02:45:12.079759 2667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:12.081454 2652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:45:12.081615 2652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-1/wal
uuid: "3bf1aa175aa149e6823c3f56d5748665"
format_stamp: "Formatted at 2026-08-06 02:45:12 on dist-test-slave-8tdl"
server_key: "ddc9f4a86c1e8c42c5d1d85c0f288cde"
server_key_iv: "6b9c2f734e33d2fa4cf7d7531fab1087"
server_key_version: "encryptionkey@0"
I20260806 02:45:12.081799 2652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:12.104681 2652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:12.109680 2652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:12.109933 2652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:12.111017 2652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:12.112514 2652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:12.112596 2652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:12.112685 2652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:12.112732 2652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:12.148801 2652 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.66:41865
I20260806 02:45:12.148893 2780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.66:41865 every 8 connection(s)
I20260806 02:45:12.150149 2652 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
I20260806 02:45:12.158896 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 2652
I20260806 02:45:12.159114 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance
I20260806 02:45:12.159662 1865 external_mini_cluster.cc:1428] Setting key f7e3de824634a668effbf2762502a6f4
Aug 06 02:45:12 dist-test-slave-8tdl krb5kdc[2407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984312, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:12.173473 2783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:12.152924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:48393 (local address 127.1.210.126:42041)
0806 02:45:12.153435 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:12.153441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:12.154509 (+ 1068us) server_negotiation.cc:407] Connection header received
0806 02:45:12.156644 (+ 2135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:12.156650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:12.156738 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:12.156871 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:12.158413 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:12.159546 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:12.161808 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:12.162399 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:12.162532 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:12.166222 (+ 3690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:12.166231 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:12.166238 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:12.166285 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:12.169607 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:12.170787 (+ 1180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:12.170792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:12.170795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:12.170930 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:12.171491 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:12.171495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:12.171497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:12.171783 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:12.171904 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:12.172971 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0806 02:45:12.173167 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":261,"threads_started":1}
I20260806 02:45:12.175369 2784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:12.153591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:42041 (local address 127.1.210.66:48393)
0806 02:45:12.154323 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:12.154368 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:12.156348 (+ 1980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:12.157123 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:12.157131 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:12.157666 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:12.158232 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:12.158248 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:12.159719 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:12.159761 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:12.161614 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:12.161631 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:12.162619 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:12.162624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:12.162899 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:12.163714 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:12.163739 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:12.165973 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:12.169771 (+ 3798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:12.169785 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:12.169809 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:12.170568 (+ 759us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:12.171060 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:12.171069 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:12.171077 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:12.171356 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:12.171951 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:12.171963 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:12.172508 (+ 545us) client_negotiation.cc:769] Sending connection context
0806 02:45:12.172874 (+ 366us) client_negotiation.cc:240] Negotiation successful
0806 02:45:12.173604 (+ 730us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":283,"threads_started":1}
I20260806 02:45:12.177175 2781 heartbeater.cc:347] Connected to a master server at 127.1.210.126:42041
I20260806 02:45:12.177534 2781 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:12.178369 2781 heartbeater.cc:511] Master 127.1.210.126:42041 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.210.67@KRBTEST.COM; defaulting to no policy
I20260806 02:45:12.180127 2453 ts_manager.cc:194] Registered new tserver with Master: 3bf1aa175aa149e6823c3f56d5748665 (127.1.210.66:41865)
I20260806 02:45:12.181061 2453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.66@KRBTEST.COM'} at 127.1.210.66:48393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab.
Entry for principal kudu/127.1.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab.
I20260806 02:45:12.207479 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.67:0
--local_ip_for_outbound_sockets=127.1.210.67
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:42041
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:36465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:12.586249 2787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:12.586651 2787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:12.586930 2787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:12.596678 2787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:12.596884 2787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:12.597154 2787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.67
I20260806 02:45:12.608780 2787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:36465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:42041
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.67
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:12.610205 2787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:12.611704 2787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:12.630712 2792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:12.631048 2795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:12.631603 2793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:12.633074 2787 server_base.cc:1034] running on GCE node
I20260806 02:45:12.633841 2787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:12.635239 2787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:12.636550 2787 hybrid_clock.cc:648] HybridClock initialized: now 1785984312636430 us; error 58 us; skew 500 ppm
I20260806 02:45:12.637032 2787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:45:12 dist-test-slave-8tdl krb5kdc[2407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984312, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:12.641167 2787 init.cc:377] Logged in from keytab as kudu/127.1.210.67@KRBTEST.COM (short username kudu)
I20260806 02:45:12.643412 2787 webserver.cc:460] Webserver started at http://127.1.210.67:44043/ using document root <none> and password file <none>
I20260806 02:45:12.644313 2787 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:12.644481 2787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:12.644829 2787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:12.648646 2787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-2/data/instance:
uuid: "20ec34f890a54293ad66109846cf3819"
format_stamp: "Formatted at 2026-08-06 02:45:12 on dist-test-slave-8tdl"
server_key: "c1721ed287c8db54a6fcafff1b5f07fd"
server_key_iv: "2432b9c1458c0ea8482708f78a3f34a2"
server_key_version: "encryptionkey@0"
I20260806 02:45:12.649701 2787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance:
uuid: "20ec34f890a54293ad66109846cf3819"
format_stamp: "Formatted at 2026-08-06 02:45:12 on dist-test-slave-8tdl"
server_key: "c1721ed287c8db54a6fcafff1b5f07fd"
server_key_iv: "2432b9c1458c0ea8482708f78a3f34a2"
server_key_version: "encryptionkey@0"
I20260806 02:45:12.656810 2787 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20260806 02:45:12.658541 2646 heartbeater.cc:503] Master 127.1.210.126:42041 was elected leader, sending a full tablet report...
I20260806 02:45:12.661468 2802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:12.663187 2787 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 02:45:12.663383 2787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-2/wal
uuid: "20ec34f890a54293ad66109846cf3819"
format_stamp: "Formatted at 2026-08-06 02:45:12 on dist-test-slave-8tdl"
server_key: "c1721ed287c8db54a6fcafff1b5f07fd"
server_key_iv: "2432b9c1458c0ea8482708f78a3f34a2"
server_key_version: "encryptionkey@0"
I20260806 02:45:12.663606 2787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:12.675835 2787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:12.680792 2787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:12.681111 2787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:12.682102 2787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:12.683579 2787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:12.683686 2787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:12.683835 2787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:12.683914 2787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:12.720227 2787 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.67:35801
I20260806 02:45:12.720270 2915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.67:35801 every 8 connection(s)
I20260806 02:45:12.721593 2787 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
I20260806 02:45:12.728274 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 2787
I20260806 02:45:12.728515 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance
I20260806 02:45:12.728994 1865 external_mini_cluster.cc:1428] Setting key eb5834f8ade2f17e8cd685d531752dd7
Aug 06 02:45:12 dist-test-slave-8tdl krb5kdc[2407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984312, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:12.744526 2918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:12.724528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:51621 (local address 127.1.210.126:42041)
0806 02:45:12.725119 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:12.725125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:12.726227 (+ 1102us) server_negotiation.cc:407] Connection header received
0806 02:45:12.728276 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:12.728282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:12.728380 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:12.728498 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:12.730289 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:12.731380 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:12.733447 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:12.733915 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:12.734052 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:12.737863 (+ 3811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:12.737870 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:12.737875 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:12.737921 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:12.740798 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:12.741682 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:12.741686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:12.741688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:12.741825 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:12.742381 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:12.742384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:12.742387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:12.742735 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:12.742878 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:12.743795 (+ 917us) server_negotiation.cc:299] Negotiation successful
0806 02:45:12.744055 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":287,"threads_started":1}
I20260806 02:45:12.745697 2919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:12.725227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:42041 (local address 127.1.210.67:51621)
0806 02:45:12.726033 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:12.726077 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:12.727987 (+ 1910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:12.728774 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:12.728783 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:12.729380 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:12.730116 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:12.730136 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:12.731552 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:12.731556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:12.732964 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:12.732979 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:12.734179 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:12.734187 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:12.734434 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:12.735265 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:12.735292 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:12.737624 (+ 2332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:12.740991 (+ 3367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:12.741000 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:12.741016 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:12.741548 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:12.742018 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:12.742022 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:12.742024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:12.742244 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:12.742898 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:12.742907 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:12.743361 (+ 454us) client_negotiation.cc:769] Sending connection context
0806 02:45:12.743636 (+ 275us) client_negotiation.cc:240] Negotiation successful
0806 02:45:12.744254 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":307,"threads_started":1}
I20260806 02:45:12.747525 2916 heartbeater.cc:347] Connected to a master server at 127.1.210.126:42041
I20260806 02:45:12.747938 2916 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:12.748759 2916 heartbeater.cc:511] Master 127.1.210.126:42041 requested a full tablet report, sending...
I20260806 02:45:12.750291 2453 ts_manager.cc:194] Registered new tserver with Master: 20ec34f890a54293ad66109846cf3819 (127.1.210.67:35801)
I20260806 02:45:12.751185 2453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.67@KRBTEST.COM'} at 127.1.210.67:51621
I20260806 02:45:12.758524 1865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:45:12.778656 2918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:12.765156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42706 (local address 127.1.210.126:42041)
0806 02:45:12.765353 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:12.765361 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:12.765804 (+ 443us) server_negotiation.cc:407] Connection header received
0806 02:45:12.766031 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:12.766035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:12.766099 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:12.766202 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:12.767872 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:12.768741 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:12.770109 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:12.770502 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:12.770597 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:12.773161 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:12.773167 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:12.773170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:12.773200 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:12.775822 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:12.776486 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:12.776492 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:12.776496 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:12.776669 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:12.777195 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:12.777199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:12.777201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:12.777540 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:12.777730 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:12.778224 (+ 494us) server_negotiation.cc:299] Negotiation successful
0806 02:45:12.778403 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 02:45:12.786127 2453 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:45:12.793251 2453 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:45:12.832902 2932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:12.806587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34448 (local address 127.1.210.66:41865)
0806 02:45:12.822497 (+ 15910us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:12.822511 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:12.822551 (+ 40us) server_negotiation.cc:407] Connection header received
0806 02:45:12.822708 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:12.822720 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:12.823090 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:12.823336 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:12.825103 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:12.826820 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:12.830124 (+ 3304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:12.831634 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:12.831776 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:12.831996 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:12.832355 (+ 359us) server_negotiation.cc:299] Negotiation successful
0806 02:45:12.832540 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15671,"thread_start_us":14400,"threads_started":1}
I20260806 02:45:12.833160 2931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:12.806146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:41865 (local address 127.0.0.1:34448)
0806 02:45:12.818912 (+ 12766us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:12.818981 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:12.819202 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:12.823717 (+ 4515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:12.823729 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:12.823770 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:12.824309 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:12.824326 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:12.826977 (+ 2651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:12.826985 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:12.829946 (+ 2961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:12.829960 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:12.831846 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:12.831852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:12.832082 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:12.832149 (+ 67us) client_negotiation.cc:769] Sending connection context
0806 02:45:12.832322 (+ 173us) client_negotiation.cc:240] Negotiation successful
0806 02:45:12.832530 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12390,"mutex_wait_us":6,"thread_start_us":19614,"threads_started":1}
I20260806 02:45:12.834364 2930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:12.806719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:35801 (local address 127.0.0.1:52866)
0806 02:45:12.826030 (+ 19311us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:12.826050 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:12.826199 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:12.826822 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:12.826827 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:12.826851 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:12.827466 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:12.827478 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:12.829520 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:12.829526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:12.831357 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:12.831371 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:12.833433 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:12.833440 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:12.833649 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:12.833688 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 02:45:12.833782 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 02:45:12.833913 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19187,"mutex_wait_us":2,"thread_start_us":13573,"threads_started":1}
I20260806 02:45:12.834728 2934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:12.807376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57234 (local address 127.1.210.65:33043)
0806 02:45:12.825797 (+ 18421us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:12.825806 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:12.825839 (+ 33us) server_negotiation.cc:407] Connection header received
0806 02:45:12.825988 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:12.825997 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:12.826309 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:12.826520 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:12.827447 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:12.828539 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:12.831209 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:12.833006 (+ 1797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:12.833164 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:12.833419 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:12.834141 (+ 722us) server_negotiation.cc:299] Negotiation successful
0806 02:45:12.834323 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18209,"thread_start_us":18003,"threads_started":1}
I20260806 02:45:12.835407 2935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:12.807657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52866 (local address 127.1.210.67:35801)
0806 02:45:12.825101 (+ 17444us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:12.825125 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:12.826172 (+ 1047us) server_negotiation.cc:407] Connection header received
0806 02:45:12.826371 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:12.826379 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:12.826603 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:12.826822 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:12.827667 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:12.829324 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:12.831515 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:12.832915 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:12.833067 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:12.833283 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:12.835021 (+ 1738us) server_negotiation.cc:299] Negotiation successful
0806 02:45:12.835160 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17204,"thread_start_us":17020,"threads_started":1}
I20260806 02:45:12.834657 2933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:12.806505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:33043 (local address 127.0.0.1:57234)
0806 02:45:12.820616 (+ 14111us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:12.820638 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:12.820767 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:12.826524 (+ 5757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:12.826529 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:12.826554 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:12.827200 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:12.827220 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:12.828673 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:12.828680 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:12.831039 (+ 2359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:12.831054 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:12.833629 (+ 2575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:12.833638 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:12.833878 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:12.833912 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 02:45:12.834081 (+ 169us) client_negotiation.cc:240] Negotiation successful
0806 02:45:12.834228 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13952,"thread_start_us":12928,"threads_started":1}
I20260806 02:45:12.837576 2715 tablet_service.cc:1467] Processing CreateTablet for tablet 12d1cb9055a64f7990293bf5c6c6ad18 (DEFAULT_TABLE table=test-table [id=501644e812f0481b90cb46ffa9d13bf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:12.838915 2715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12d1cb9055a64f7990293bf5c6c6ad18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:12.839496 2578 tablet_service.cc:1467] Processing CreateTablet for tablet 12d1cb9055a64f7990293bf5c6c6ad18 (DEFAULT_TABLE table=test-table [id=501644e812f0481b90cb46ffa9d13bf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:12.840673 2578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12d1cb9055a64f7990293bf5c6c6ad18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:12.843585 2850 tablet_service.cc:1467] Processing CreateTablet for tablet 12d1cb9055a64f7990293bf5c6c6ad18 (DEFAULT_TABLE table=test-table [id=501644e812f0481b90cb46ffa9d13bf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:12.845364 2850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12d1cb9055a64f7990293bf5c6c6ad18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:12.852536 2936 tablet_bootstrap.cc:492] T 12d1cb9055a64f7990293bf5c6c6ad18 P 474296ae66f84e53b4fb9582c4fd4493: Bootstrap starting.
I20260806 02:45:12.855448 2937 tablet_bootstrap.cc:492] T 12d1cb9055a64f7990293bf5c6c6ad18 P 3bf1aa175aa149e6823c3f56d5748665: Bootstrap starting.
I20260806 02:45:12.856866 2936 tablet_bootstrap.cc:654] T 12d1cb9055a64f7990293bf5c6c6ad18 P 474296ae66f84e53b4fb9582c4fd4493: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:12.858253 2936 log.cc:824] T 12d1cb9055a64f7990293bf5c6c6ad18 P 474296ae66f84e53b4fb9582c4fd4493: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:12.859411 2937 tablet_bootstrap.cc:654] T 12d1cb9055a64f7990293bf5c6c6ad18 P 3bf1aa175aa149e6823c3f56d5748665: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:12.860792 2937 log.cc:824] T 12d1cb9055a64f7990293bf5c6c6ad18 P 3bf1aa175aa149e6823c3f56d5748665: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:12.861780 2936 tablet_bootstrap.cc:492] T 12d1cb9055a64f7990293bf5c6c6ad18 P 474296ae66f84e53b4fb9582c4fd4493: No bootstrap required, opened a new log
I20260806 02:45:12.862079 2936 ts_tablet_manager.cc:1397] T 12d1cb9055a64f7990293bf5c6c6ad18 P 474296ae66f84e53b4fb9582c4fd4493: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260806 02:45:12.864384 2937 tablet_bootstrap.cc:492] T 12d1cb9055a64f7990293bf5c6c6ad18 P 3bf1aa175aa149e6823c3f56d5748665: No bootstrap required, opened a new log
I20260806 02:45:12.864683 2937 ts_tablet_manager.cc:1397] T 12d1cb9055a64f7990293bf5c6c6ad18 P 3bf1aa175aa149e6823c3f56d5748665: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260806 02:45:12.865373 2939 tablet_bootstrap.cc:492] T 12d1cb9055a64f7990293bf5c6c6ad18 P 20ec34f890a54293ad66109846cf3819: Bootstrap starting.
I20260806 02:45:12.869437 2939 tablet_bootstrap.cc:654] T 12d1cb9055a64f7990293bf5c6c6ad18 P 20ec34f890a54293ad66109846cf3819: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:12.870929 2939 log.cc:824] T 12d1cb9055a64f7990293bf5c6c6ad18 P 20ec34f890a54293ad66109846cf3819: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:12.873652 2939 tablet_bootstrap.cc:492] T 12d1cb9055a64f7990293bf5c6c6ad18 P 20ec34f890a54293ad66109846cf3819: No bootstrap required, opened a new log
I20260806 02:45:12.873327 2936 raft_consensus.cc:348] T 12d1cb9055a64f7990293bf5c6c6ad18 P 474296ae66f84e53b4fb9582c4fd4493 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bf1aa175aa149e6823c3f56d5748665" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 41865 } } peers { permanent_uuid: "474296ae66f84e53b4fb9582c4fd4493" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33043 } } peers { permanent_uuid: "20ec34f890a54293ad66109846cf3819" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 35801 } }
I20260806 02:45:12.873808 2936 raft_consensus.cc:374] T 12d1cb9055a64f7990293bf5c6c6ad18 P 474296ae66f84e53b4fb9582c4fd4493 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:12.873926 2936 raft_consensus.cc:729] T 12d1cb9055a64f7990293bf5c6c6ad18 P 474296ae66f84e53b4fb9582c4fd4493 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 474296ae66f84e53b4fb9582c4fd4493, State: Initialized, Role: FOLLOWER
I20260806 02:45:12.873967 2939 ts_tablet_manager.cc:1397] T 12d1cb9055a64f7990293bf5c6c6ad18 P 20ec34f890a54293ad66109846cf3819: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260806 02:45:12.874594 2936 consensus_queue.cc:260] T 12d1cb9055a64f7990293bf5c6c6ad18 P 474296ae66f84e53b4fb9582c4fd4493 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bf1aa175aa149e6823c3f56d5748665" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 41865 } } peers { permanent_uuid: "474296ae66f84e53b4fb9582c4fd4493" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33043 } } peers { permanent_uuid: "20ec34f890a54293ad66109846cf3819" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 35801 } }
I20260806 02:45:12.875535 2937 raft_consensus.cc:348] T 12d1cb9055a64f7990293bf5c6c6ad18 P 3bf1aa175aa149e6823c3f56d5748665 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bf1aa175aa149e6823c3f56d5748665" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 41865 } } peers { permanent_uuid: "474296ae66f84e53b4fb9582c4fd4493" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33043 } } peers { permanent_uuid: "20ec34f890a54293ad66109846cf3819" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 35801 } }
I20260806 02:45:12.875985 2937 raft_consensus.cc:374] T 12d1cb9055a64f7990293bf5c6c6ad18 P 3bf1aa175aa149e6823c3f56d5748665 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:12.876096 2937 raft_consensus.cc:729] T 12d1cb9055a64f7990293bf5c6c6ad18 P 3bf1aa175aa149e6823c3f56d5748665 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bf1aa175aa149e6823c3f56d5748665, State: Initialized, Role: FOLLOWER
I20260806 02:45:12.876653 2936 ts_tablet_manager.cc:1428] T 12d1cb9055a64f7990293bf5c6c6ad18 P 474296ae66f84e53b4fb9582c4fd4493: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
I20260806 02:45:12.876755 2937 consensus_queue.cc:260] T 12d1cb9055a64f7990293bf5c6c6ad18 P 3bf1aa175aa149e6823c3f56d5748665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bf1aa175aa149e6823c3f56d5748665" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 41865 } } peers { permanent_uuid: "474296ae66f84e53b4fb9582c4fd4493" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33043 } } peers { permanent_uuid: "20ec34f890a54293ad66109846cf3819" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 35801 } }
I20260806 02:45:12.878131 2781 heartbeater.cc:503] Master 127.1.210.126:42041 was elected leader, sending a full tablet report...
I20260806 02:45:12.878539 2937 ts_tablet_manager.cc:1428] T 12d1cb9055a64f7990293bf5c6c6ad18 P 3bf1aa175aa149e6823c3f56d5748665: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260806 02:45:12.884454 2939 raft_consensus.cc:348] T 12d1cb9055a64f7990293bf5c6c6ad18 P 20ec34f890a54293ad66109846cf3819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bf1aa175aa149e6823c3f56d5748665" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 41865 } } peers { permanent_uuid: "474296ae66f84e53b4fb9582c4fd4493" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33043 } } peers { permanent_uuid: "20ec34f890a54293ad66109846cf3819" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 35801 } }
I20260806 02:45:12.884814 2939 raft_consensus.cc:374] T 12d1cb9055a64f7990293bf5c6c6ad18 P 20ec34f890a54293ad66109846cf3819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:12.884892 2939 raft_consensus.cc:729] T 12d1cb9055a64f7990293bf5c6c6ad18 P 20ec34f890a54293ad66109846cf3819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20ec34f890a54293ad66109846cf3819, State: Initialized, Role: FOLLOWER
I20260806 02:45:12.885557 2939 consensus_queue.cc:260] T 12d1cb9055a64f7990293bf5c6c6ad18 P 20ec34f890a54293ad66109846cf3819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bf1aa175aa149e6823c3f56d5748665" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 41865 } } peers { permanent_uuid: "474296ae66f84e53b4fb9582c4fd4493" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33043 } } peers { permanent_uuid: "20ec34f890a54293ad66109846cf3819" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 35801 } }
I20260806 02:45:12.886775 2916 heartbeater.cc:503] Master 127.1.210.126:42041 was elected leader, sending a full tablet report...
I20260806 02:45:12.887429 2939 ts_tablet_manager.cc:1428] T 12d1cb9055a64f7990293bf5c6c6ad18 P 20ec34f890a54293ad66109846cf3819: Time spent starting tablet: real 0.013s user 0.012s sys 0.002s
W20260806 02:45:12.902384 2782 tablet.cc:2369] T 12d1cb9055a64f7990293bf5c6c6ad18 P 3bf1aa175aa149e6823c3f56d5748665: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:45:12.973384 2917 tablet.cc:2369] T 12d1cb9055a64f7990293bf5c6c6ad18 P 20ec34f890a54293ad66109846cf3819: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:45:12.992385 2943 raft_consensus.cc:482] T 12d1cb9055a64f7990293bf5c6c6ad18 P 3bf1aa175aa149e6823c3f56d5748665 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:45:12.992719 2943 raft_consensus.cc:504] T 12d1cb9055a64f7990293bf5c6c6ad18 P 3bf1aa175aa149e6823c3f56d5748665 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bf1aa175aa149e6823c3f56d5748665" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 41865 } } peers { permanent_uuid: "474296ae66f84e53b4fb9582c4fd4493" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33043 } } peers { permanent_uuid: "20ec34f890a54293ad66109846cf3819" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 35801 } }
I20260806 02:45:12.994926 2943 leader_election.cc:288] T 12d1cb9055a64f7990293bf5c6c6ad18 P 3bf1aa175aa149e6823c3f56d5748665 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 474296ae66f84e53b4fb9582c4fd4493 (127.1.210.65:33043), 20ec34f890a54293ad66109846cf3819 (127.1.210.67:35801)
I20260806 02:45:13.002470 2934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:12.995683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:48759 (local address 127.1.210.65:33043)
0806 02:45:12.995862 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:12.995867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:12.996496 (+ 629us) server_negotiation.cc:407] Connection header received
0806 02:45:12.996825 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:12.996835 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:12.996927 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:12.997021 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:12.997603 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:12.998743 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:13.000719 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:13.001501 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:13.001701 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:13.001784 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:13.002086 (+ 302us) server_negotiation.cc:299] Negotiation successful
0806 02:45:13.002217 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 02:45:13.002662 2946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:12.995190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:33043 (local address 127.1.210.66:48759)
0806 02:45:12.996369 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:12.996395 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:12.996629 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:12.997053 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:12.997058 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:12.997084 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:12.997447 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:12.997458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:12.998877 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:12.998882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:13.000544 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:13.000557 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:13.001733 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:13.001739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:13.001915 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:13.001944 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 02:45:13.002056 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 02:45:13.002192 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1040,"thread_start_us":1261,"threads_started":1}
I20260806 02:45:13.003791 2600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12d1cb9055a64f7990293bf5c6c6ad18" candidate_uuid: "3bf1aa175aa149e6823c3f56d5748665" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "474296ae66f84e53b4fb9582c4fd4493" is_pre_election: true
I20260806 02:45:13.004084 2935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:12.995448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:54715 (local address 127.1.210.67:35801)
0806 02:45:12.995627 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:12.995635 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:12.996924 (+ 1289us) server_negotiation.cc:407] Connection header received
0806 02:45:12.997149 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:12.997158 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:12.997246 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:12.997315 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:12.998135 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:12.999247 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:13.001785 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:13.003061 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:13.003224 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:13.003296 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:13.003687 (+ 391us) server_negotiation.cc:299] Negotiation successful
0806 02:45:13.003829 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 02:45:13.004302 2945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:12.995250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:35801 (local address 127.1.210.66:54715)
0806 02:45:12.996794 (+ 1544us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:12.996812 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:12.996955 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:12.997529 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:12.997533 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:12.997558 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:12.997924 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:12.997950 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:12.999405 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:12.999411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:13.001597 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:13.001611 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:13.003286 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:13.003293 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:13.003496 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:13.003537 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 02:45:13.003641 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 02:45:13.003792 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1406,"mutex_wait_us":4,"thread_start_us":766,"threads_started":1}
I20260806 02:45:13.004451 2600 raft_consensus.cc:2454] T 12d1cb9055a64f7990293bf5c6c6ad18 P 474296ae66f84e53b4fb9582c4fd4493 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bf1aa175aa149e6823c3f56d5748665 in term 0.
I20260806 02:45:13.005251 2671 leader_election.cc:302] T 12d1cb9055a64f7990293bf5c6c6ad18 P 3bf1aa175aa149e6823c3f56d5748665 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bf1aa175aa149e6823c3f56d5748665, 474296ae66f84e53b4fb9582c4fd4493; no voters:
I20260806 02:45:13.005692 2943 raft_consensus.cc:2781] T 12d1cb9055a64f7990293bf5c6c6ad18 P 3bf1aa175aa149e6823c3f56d5748665 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:45:13.005604 2870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12d1cb9055a64f7990293bf5c6c6ad18" candidate_uuid: "3bf1aa175aa149e6823c3f56d5748665" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20ec34f890a54293ad66109846cf3819" is_pre_election: true
I20260806 02:45:13.005856 2943 raft_consensus.cc:482] T 12d1cb9055a64f7990293bf5c6c6ad18 P 3bf1aa175aa149e6823c3f56d5748665 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:45:13.005960 2943 raft_consensus.cc:3037] T 12d1cb9055a64f7990293bf5c6c6ad18 P 3bf1aa175aa149e6823c3f56d5748665 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:13.006155 2870 raft_consensus.cc:2454] T 12d1cb9055a64f7990293bf5c6c6ad18 P 20ec34f890a54293ad66109846cf3819 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bf1aa175aa149e6823c3f56d5748665 in term 0.
I20260806 02:45:13.008548 2943 raft_consensus.cc:504] T 12d1cb9055a64f7990293bf5c6c6ad18 P 3bf1aa175aa149e6823c3f56d5748665 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bf1aa175aa149e6823c3f56d5748665" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 41865 } } peers { permanent_uuid: "474296ae66f84e53b4fb9582c4fd4493" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33043 } } peers { permanent_uuid: "20ec34f890a54293ad66109846cf3819" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 35801 } }
I20260806 02:45:13.009267 2943 leader_election.cc:288] T 12d1cb9055a64f7990293bf5c6c6ad18 P 3bf1aa175aa149e6823c3f56d5748665 [CANDIDATE]: Term 1 election: Requested vote from peers 474296ae66f84e53b4fb9582c4fd4493 (127.1.210.65:33043), 20ec34f890a54293ad66109846cf3819 (127.1.210.67:35801)
I20260806 02:45:13.009882 2600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12d1cb9055a64f7990293bf5c6c6ad18" candidate_uuid: "3bf1aa175aa149e6823c3f56d5748665" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "474296ae66f84e53b4fb9582c4fd4493"
I20260806 02:45:13.009975 2870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12d1cb9055a64f7990293bf5c6c6ad18" candidate_uuid: "3bf1aa175aa149e6823c3f56d5748665" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20ec34f890a54293ad66109846cf3819"
I20260806 02:45:13.010156 2600 raft_consensus.cc:3037] T 12d1cb9055a64f7990293bf5c6c6ad18 P 474296ae66f84e53b4fb9582c4fd4493 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:13.010221 2870 raft_consensus.cc:3037] T 12d1cb9055a64f7990293bf5c6c6ad18 P 20ec34f890a54293ad66109846cf3819 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:13.012585 2870 raft_consensus.cc:2454] T 12d1cb9055a64f7990293bf5c6c6ad18 P 20ec34f890a54293ad66109846cf3819 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bf1aa175aa149e6823c3f56d5748665 in term 1.
I20260806 02:45:13.012706 2600 raft_consensus.cc:2454] T 12d1cb9055a64f7990293bf5c6c6ad18 P 474296ae66f84e53b4fb9582c4fd4493 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bf1aa175aa149e6823c3f56d5748665 in term 1.
I20260806 02:45:13.013121 2670 leader_election.cc:302] T 12d1cb9055a64f7990293bf5c6c6ad18 P 3bf1aa175aa149e6823c3f56d5748665 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20ec34f890a54293ad66109846cf3819, 3bf1aa175aa149e6823c3f56d5748665; no voters:
I20260806 02:45:13.013531 2943 raft_consensus.cc:2781] T 12d1cb9055a64f7990293bf5c6c6ad18 P 3bf1aa175aa149e6823c3f56d5748665 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:13.014012 2943 raft_consensus.cc:686] T 12d1cb9055a64f7990293bf5c6c6ad18 P 3bf1aa175aa149e6823c3f56d5748665 [term 1 LEADER]: Becoming Leader. State: Replica: 3bf1aa175aa149e6823c3f56d5748665, State: Running, Role: LEADER
I20260806 02:45:13.014504 2943 consensus_queue.cc:237] T 12d1cb9055a64f7990293bf5c6c6ad18 P 3bf1aa175aa149e6823c3f56d5748665 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bf1aa175aa149e6823c3f56d5748665" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 41865 } } peers { permanent_uuid: "474296ae66f84e53b4fb9582c4fd4493" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33043 } } peers { permanent_uuid: "20ec34f890a54293ad66109846cf3819" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 35801 } }
I20260806 02:45:13.019887 2453 catalog_manager.cc:5495] T 12d1cb9055a64f7990293bf5c6c6ad18 P 3bf1aa175aa149e6823c3f56d5748665 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3bf1aa175aa149e6823c3f56d5748665 (127.1.210.66). New cstate: current_term: 1 leader_uuid: "3bf1aa175aa149e6823c3f56d5748665" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bf1aa175aa149e6823c3f56d5748665" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 41865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "474296ae66f84e53b4fb9582c4fd4493" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20ec34f890a54293ad66109846cf3819" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 35801 } health_report { overall_health: UNKNOWN } } }
Aug 06 02:45:13 dist-test-slave-8tdl krb5kdc[2407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.65@KRBTEST.COM
I20260806 02:45:13.067386 2934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:13.047387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57250 (local address 127.1.210.65:33043)
0806 02:45:13.047688 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:13.047698 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:13.048299 (+ 601us) server_negotiation.cc:407] Connection header received
0806 02:45:13.048593 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:13.048597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:13.048706 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:13.048810 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:13.050585 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:13.051565 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:13.053342 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:13.053782 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:13.053889 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:13.059156 (+ 5267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:13.059174 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:13.059191 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:13.059229 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:13.063684 (+ 4455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:13.064584 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:13.064592 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:13.064595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:13.064832 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:13.065456 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:13.065462 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:13.065464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:13.065933 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:13.066289 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:13.066523 (+ 234us) server_negotiation.cc:299] Negotiation successful
0806 02:45:13.066766 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Aug 06 02:45:13 dist-test-slave-8tdl krb5kdc[2407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984313, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 02:45:13 dist-test-slave-8tdl krb5kdc[2407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984313, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.210.65@KRBTEST.COM
I20260806 02:45:13.124394 2934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:13.106808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57266 (local address 127.1.210.65:33043)
0806 02:45:13.107111 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:13.107120 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:13.107301 (+ 181us) server_negotiation.cc:407] Connection header received
0806 02:45:13.107555 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:13.107559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:13.107624 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:13.107731 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:13.109557 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:13.110380 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:13.112138 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:13.112590 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:13.112737 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:13.117079 (+ 4342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:13.117085 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:13.117089 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:13.117127 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:13.121040 (+ 3913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:13.121928 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:13.121933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:13.121936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:13.122243 (+ 307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:13.122776 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:13.122780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:13.122782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:13.123243 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:13.123628 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:13.123736 (+ 108us) server_negotiation.cc:299] Negotiation successful
0806 02:45:13.123951 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260806 02:45:13.126446 2647 tablet.cc:2369] T 12d1cb9055a64f7990293bf5c6c6ad18 P 474296ae66f84e53b4fb9582c4fd4493: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 02:45:13 dist-test-slave-8tdl krb5kdc[2407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 02:45:13 dist-test-slave-8tdl krb5kdc[2407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.65@KRBTEST.COM
I20260806 02:45:13.185925 2934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:13.166681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57278 (local address 127.1.210.65:33043)
0806 02:45:13.166964 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:13.166971 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:13.167196 (+ 225us) server_negotiation.cc:407] Connection header received
0806 02:45:13.167438 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:13.167443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:13.167536 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:13.167650 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:13.169566 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:13.170274 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:13.172124 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:13.172583 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:13.172729 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:13.178198 (+ 5469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:13.178204 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:13.178209 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:13.178253 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:13.182576 (+ 4323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:13.183480 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:13.183484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:13.183487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:13.183678 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:13.184149 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:13.184154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:13.184156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:13.184626 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:13.184840 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:13.185226 (+ 386us) server_negotiation.cc:299] Negotiation successful
0806 02:45:13.185479 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260806 02:45:13.188338 2558 tablet_service.cc:2908] Rejecting scan request for tablet 12d1cb9055a64f7990293bf5c6c6ad18: Uninitialized: clean time has not yet been initialized
I20260806 02:45:13.192185 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 2515
I20260806 02:45:13.208726 2934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:13.193139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57292 (local address 127.1.210.65:33043)
0806 02:45:13.193368 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:13.193377 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:13.193778 (+ 401us) server_negotiation.cc:407] Connection header received
0806 02:45:13.193941 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:13.193945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:13.194028 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:13.194112 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:13.195945 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:13.196646 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:13.198316 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:13.198824 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:13.198934 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:13.202074 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:13.202079 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:13.202082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:13.202126 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:13.205254 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:13.206129 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:13.206134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:13.206136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:13.206359 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:13.206944 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:13.206948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:13.206951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:13.207420 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:13.207795 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:13.207979 (+ 184us) server_negotiation.cc:299] Negotiation successful
0806 02:45:13.208188 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 02:45:13.209468 2641 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:45:13.305193 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 2515
I20260806 02:45:13.324674 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 2652
Aug 06 02:45:13 dist-test-slave-8tdl krb5kdc[2407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.66@KRBTEST.COM
I20260806 02:45:13.342424 2932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:13.325753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34450 (local address 127.1.210.66:41865)
0806 02:45:13.325966 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:13.325971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:13.325991 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:45:13.326189 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:13.326194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:13.326277 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:13.326375 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:13.327771 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:13.328921 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:13.330396 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:13.331287 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:13.331407 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:13.336077 (+ 4670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:13.336103 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:13.336129 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:13.336179 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:13.339357 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:13.340127 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:13.340135 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:13.340137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:13.340336 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:13.340739 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:13.340744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:13.340746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:13.341155 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:13.341391 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:13.341797 (+ 406us) server_negotiation.cc:299] Negotiation successful
0806 02:45:13.342046 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 02:45:13.343238 2776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:45:13.439810 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 2652
I20260806 02:45:13.457358 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 2787
Aug 06 02:45:13 dist-test-slave-8tdl krb5kdc[2407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.67@KRBTEST.COM
I20260806 02:45:13.475564 2935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:13.458421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52876 (local address 127.1.210.67:35801)
0806 02:45:13.458653 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:13.458658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:13.458716 (+ 58us) server_negotiation.cc:407] Connection header received
0806 02:45:13.458808 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:13.458812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:13.458880 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:13.459016 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:13.460413 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:13.461151 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:13.462507 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:13.463070 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:13.463222 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:13.469104 (+ 5882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:13.469121 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:13.469140 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:13.469191 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:13.472419 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:13.473268 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:13.473277 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:13.473279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:13.473442 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:13.473876 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:13.473881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:13.473883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:13.474356 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:13.474602 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:13.475091 (+ 489us) server_negotiation.cc:299] Negotiation successful
0806 02:45:13.475271 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 02:45:13.476392 2911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:45:13.567587 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 2787
I20260806 02:45:13.586371 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 2420
I20260806 02:45:13.587980 2483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:45:13.700707 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 2420
2026-08-06T02:45:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4529 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:45:13 dist-test-slave-8tdl krb5kdc[2966](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:45:13 dist-test-slave-8tdl krb5kdc[2966](info): set up 2 sockets
Aug 06 02:45:13 dist-test-slave-8tdl krb5kdc[2966](info): commencing operation
krb5kdc: starting...
W20260806 02:45:14.801792 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:45:14 dist-test-slave-8tdl krb5kdc[2966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:45:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
I20260806 02:45:15.011868 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:37725
--webserver_interface=127.1.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:42969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:37725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:15.350613 2979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:15.351019 2979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:15.351116 2979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:15.358664 2979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:15.358880 2979 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 02:45:15.358922 2979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:15.358989 2979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:15.359043 2979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:15.359107 2979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:45:15.369712 2979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:42969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:37725
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:37725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:15.371063 2979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:15.372396 2979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:15.385239 2987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:15.385305 2985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:15.385308 2984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:15.386873 2979 server_base.cc:1034] running on GCE node
I20260806 02:45:15.387781 2979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:15.389367 2979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:15.390635 2979 hybrid_clock.cc:648] HybridClock initialized: now 1785984315390556 us; error 71 us; skew 500 ppm
I20260806 02:45:15.391036 2979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:45:15 dist-test-slave-8tdl krb5kdc[2966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984315, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:15.394498 2979 init.cc:377] Logged in from keytab as kudu/127.1.210.126@KRBTEST.COM (short username kudu)
I20260806 02:45:15.396061 2979 webserver.cc:460] Webserver started at http://127.1.210.126:46577/ using document root <none> and password file <none>
I20260806 02:45:15.396836 2979 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:15.396982 2979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:15.397359 2979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:15.400774 2979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "69e101b4d0f249a4b5cf7a908e31be9c"
format_stamp: "Formatted at 2026-08-06 02:45:15 on dist-test-slave-8tdl"
server_key: "71be98750ca31be2bb6fe25dca02b549"
server_key_iv: "d04a52f245d49d0ccfe40672623c6d43"
server_key_version: "encryptionkey@0"
I20260806 02:45:15.401702 2979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "69e101b4d0f249a4b5cf7a908e31be9c"
format_stamp: "Formatted at 2026-08-06 02:45:15 on dist-test-slave-8tdl"
server_key: "71be98750ca31be2bb6fe25dca02b549"
server_key_iv: "d04a52f245d49d0ccfe40672623c6d43"
server_key_version: "encryptionkey@0"
I20260806 02:45:15.407646 2979 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 02:45:15.411587 2994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:15.413196 2979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:45:15.413415 2979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "69e101b4d0f249a4b5cf7a908e31be9c"
format_stamp: "Formatted at 2026-08-06 02:45:15 on dist-test-slave-8tdl"
server_key: "71be98750ca31be2bb6fe25dca02b549"
server_key_iv: "d04a52f245d49d0ccfe40672623c6d43"
server_key_version: "encryptionkey@0"
I20260806 02:45:15.413671 2979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:15.432544 2979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:15.437842 2979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:15.438247 2979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:15.463028 2979 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:37725
I20260806 02:45:15.463083 3057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:37725 every 8 connection(s)
I20260806 02:45:15.464663 2979 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:45:15.468818 3058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:15.473173 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 2979
I20260806 02:45:15.473446 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:45:15.474133 1865 external_mini_cluster.cc:1428] Setting key 5b94b25f268931c89145c877e0289f63
I20260806 02:45:15.480614 3058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69e101b4d0f249a4b5cf7a908e31be9c: Bootstrap starting.
I20260806 02:45:15.484627 3058 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69e101b4d0f249a4b5cf7a908e31be9c: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:15.486016 3058 log.cc:824] T 00000000000000000000000000000000 P 69e101b4d0f249a4b5cf7a908e31be9c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:15.490113 3058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69e101b4d0f249a4b5cf7a908e31be9c: No bootstrap required, opened a new log
Aug 06 02:45:15 dist-test-slave-8tdl krb5kdc[2966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:15.498873 3058 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69e101b4d0f249a4b5cf7a908e31be9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e101b4d0f249a4b5cf7a908e31be9c" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 37725 } }
I20260806 02:45:15.499327 3058 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69e101b4d0f249a4b5cf7a908e31be9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:15.499470 3058 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69e101b4d0f249a4b5cf7a908e31be9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69e101b4d0f249a4b5cf7a908e31be9c, State: Initialized, Role: FOLLOWER
I20260806 02:45:15.500172 3058 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69e101b4d0f249a4b5cf7a908e31be9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e101b4d0f249a4b5cf7a908e31be9c" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 37725 } }
I20260806 02:45:15.500473 3058 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69e101b4d0f249a4b5cf7a908e31be9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:45:15.500643 3058 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69e101b4d0f249a4b5cf7a908e31be9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:45:15.500837 3058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69e101b4d0f249a4b5cf7a908e31be9c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:15.501145 3061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:15.476556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60078 (local address 127.1.210.126:37725)
0806 02:45:15.477436 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:15.477450 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:15.477501 (+ 51us) server_negotiation.cc:407] Connection header received
0806 02:45:15.479508 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:15.479536 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:15.479983 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:15.480328 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:15.482261 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:15.483942 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:15.485826 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:15.487179 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:15.487447 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:15.492584 (+ 5137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:15.492595 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:15.492613 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:15.492668 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:15.496296 (+ 3628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:15.497144 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:15.497152 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:15.497159 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:15.497354 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:15.497794 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:15.497800 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:15.497810 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:15.498435 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:15.498797 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:15.499296 (+ 499us) server_negotiation.cc:299] Negotiation successful
0806 02:45:15.499788 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":393,"threads_started":1}
I20260806 02:45:15.503870 3058 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69e101b4d0f249a4b5cf7a908e31be9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e101b4d0f249a4b5cf7a908e31be9c" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 37725 } }
I20260806 02:45:15.504555 3058 leader_election.cc:302] T 00000000000000000000000000000000 P 69e101b4d0f249a4b5cf7a908e31be9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69e101b4d0f249a4b5cf7a908e31be9c; no voters:
I20260806 02:45:15.505198 3058 leader_election.cc:288] T 00000000000000000000000000000000 P 69e101b4d0f249a4b5cf7a908e31be9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:45:15.505330 3063 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69e101b4d0f249a4b5cf7a908e31be9c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:15.506181 3063 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69e101b4d0f249a4b5cf7a908e31be9c [term 1 LEADER]: Becoming Leader. State: Replica: 69e101b4d0f249a4b5cf7a908e31be9c, State: Running, Role: LEADER
I20260806 02:45:15.506808 3063 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69e101b4d0f249a4b5cf7a908e31be9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e101b4d0f249a4b5cf7a908e31be9c" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 37725 } }
I20260806 02:45:15.507575 3058 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69e101b4d0f249a4b5cf7a908e31be9c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:15.510095 3064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69e101b4d0f249a4b5cf7a908e31be9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69e101b4d0f249a4b5cf7a908e31be9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e101b4d0f249a4b5cf7a908e31be9c" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 37725 } } }
I20260806 02:45:15.509974 3065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69e101b4d0f249a4b5cf7a908e31be9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69e101b4d0f249a4b5cf7a908e31be9c. Latest consensus state: current_term: 1 leader_uuid: "69e101b4d0f249a4b5cf7a908e31be9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e101b4d0f249a4b5cf7a908e31be9c" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 37725 } } }
I20260806 02:45:15.510457 3064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69e101b4d0f249a4b5cf7a908e31be9c [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:15.510457 3065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69e101b4d0f249a4b5cf7a908e31be9c [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:15.517264 3071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:15.523193 3071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:45:15.532924 3071 catalog_manager.cc:1347] Generated new cluster ID: 8d9a815ea90a4d74b45beac228d917ac
I20260806 02:45:15.533089 3071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:15.553778 3071 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:45:15.555909 3071 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:15.569101 3071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69e101b4d0f249a4b5cf7a908e31be9c: Generated new TSK 0
I20260806 02:45:15.570103 3071 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab.
Entry for principal kudu/127.1.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab.
I20260806 02:45:15.621325 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.65:0
--local_ip_for_outbound_sockets=127.1.210.65
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:37725
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:42969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:15.971452 3084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:15.971834 3084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:15.972013 3084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:15.979898 3084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:15.980062 3084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:15.980197 3084 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:45:15.980291 3084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.65
I20260806 02:45:15.990593 3084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:42969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.210.126:37725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.65
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:15.992192 3084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:15.993593 3084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:16.008852 3090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:16.008884 3092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:16.009258 3089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:16.011125 3084 server_base.cc:1034] running on GCE node
I20260806 02:45:16.011909 3084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:16.013054 3084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:16.014333 3084 hybrid_clock.cc:648] HybridClock initialized: now 1785984316014232 us; error 83 us; skew 500 ppm
I20260806 02:45:16.014761 3084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:45:16 dist-test-slave-8tdl krb5kdc[2966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984316, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:16.023275 3084 init.cc:377] Logged in from keytab as kudu/127.1.210.65@KRBTEST.COM (short username kudu)
I20260806 02:45:16.025079 3084 webserver.cc:460] Webserver started at http://127.1.210.65:39689/ using document root <none> and password file <none>
I20260806 02:45:16.025910 3084 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:16.026032 3084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:16.026396 3084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:16.029546 3084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-0/data/instance:
uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d"
format_stamp: "Formatted at 2026-08-06 02:45:16 on dist-test-slave-8tdl"
server_key: "3b9c55ba9ff6be461697035dba111e74"
server_key_iv: "8d9a1c1439f672cb1e54217395eb98f7"
server_key_version: "encryptionkey@0"
I20260806 02:45:16.030457 3084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance:
uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d"
format_stamp: "Formatted at 2026-08-06 02:45:16 on dist-test-slave-8tdl"
server_key: "3b9c55ba9ff6be461697035dba111e74"
server_key_iv: "8d9a1c1439f672cb1e54217395eb98f7"
server_key_version: "encryptionkey@0"
I20260806 02:45:16.036152 3084 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260806 02:45:16.040225 3099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:16.041930 3084 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:45:16.042145 3084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-0/wal
uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d"
format_stamp: "Formatted at 2026-08-06 02:45:16 on dist-test-slave-8tdl"
server_key: "3b9c55ba9ff6be461697035dba111e74"
server_key_iv: "8d9a1c1439f672cb1e54217395eb98f7"
server_key_version: "encryptionkey@0"
I20260806 02:45:16.042392 3084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:16.060545 3084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:16.065217 3084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:16.065539 3084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:16.068583 3084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:16.068719 3084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:16.068881 3084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:16.068953 3084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:45:16 dist-test-slave-8tdl krb5kdc[2966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984316, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:16.096684 3108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:16.069448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:52035 (local address 127.1.210.126:37725)
0806 02:45:16.070143 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:16.070151 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:16.071563 (+ 1412us) server_negotiation.cc:407] Connection header received
0806 02:45:16.074858 (+ 3295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:16.074877 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:16.075020 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:16.075175 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:16.077616 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:16.078711 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:16.081398 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:16.081915 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:16.082081 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:16.088169 (+ 6088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:16.088179 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:16.088185 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:16.088238 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:16.092074 (+ 3836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:16.093367 (+ 1293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:16.093377 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:16.093383 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:16.093578 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:16.094373 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:16.094380 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:16.094383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:16.094771 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:16.094949 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:16.096036 (+ 1087us) server_negotiation.cc:299] Negotiation successful
0806 02:45:16.096285 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":403,"threads_started":1}
I20260806 02:45:16.097972 3109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:16.070190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:37725 (local address 127.1.210.65:52035)
0806 02:45:16.071354 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:16.071409 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:16.074383 (+ 2974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:16.075516 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:16.075534 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:16.076482 (+ 948us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:16.077405 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:16.077431 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:16.078899 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:16.078903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:16.081208 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:16.081224 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:16.082240 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:16.082251 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:16.082660 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:16.083757 (+ 1097us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:16.083798 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:16.087820 (+ 4022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:16.092271 (+ 4451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:16.092282 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:16.092312 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:16.093189 (+ 877us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:16.093745 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:16.093755 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:16.093760 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:16.094052 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:16.094923 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:16.094931 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:16.095540 (+ 609us) client_negotiation.cc:769] Sending connection context
0806 02:45:16.095942 (+ 402us) client_negotiation.cc:240] Negotiation successful
0806 02:45:16.096554 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"thread_start_us":460,"threads_started":1}
I20260806 02:45:16.121838 3190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:16.105158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:37725 (local address 127.1.210.65:53643)
0806 02:45:16.105854 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:16.105879 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:16.106121 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:16.106483 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:16.106488 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:16.107110 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:16.107845 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:16.107858 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:16.109286 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:16.109292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:16.110851 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:16.110864 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:16.111927 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:16.111931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:16.112070 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:16.112992 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:16.113010 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:16.114429 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:16.118169 (+ 3740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:16.118178 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:16.118185 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:16.119026 (+ 841us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:16.119621 (+ 595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:16.119627 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:16.119634 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:16.119860 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:16.120630 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:16.120634 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:16.120921 (+ 287us) client_negotiation.cc:769] Sending connection context
0806 02:45:16.121198 (+ 277us) client_negotiation.cc:240] Negotiation successful
0806 02:45:16.121492 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":361,"threads_started":1}
I20260806 02:45:16.121932 3108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:16.105152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:53643 (local address 127.1.210.126:37725)
0806 02:45:16.105408 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:16.105417 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:16.105984 (+ 567us) server_negotiation.cc:407] Connection header received
0806 02:45:16.106273 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:16.106278 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:16.106355 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:16.106458 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:16.108079 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:16.109015 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:16.111026 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:16.111468 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:16.111647 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:16.114845 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:16.114851 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:16.114854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:16.114902 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:16.117947 (+ 3045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:16.119268 (+ 1321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:16.119275 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:16.119280 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:16.119478 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:16.120013 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:16.120019 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:16.120022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:16.120463 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:16.120664 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:16.121268 (+ 604us) server_negotiation.cc:299] Negotiation successful
0806 02:45:16.121527 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 02:45:16.123705 3220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.65:33949 every 8 connection(s)
I20260806 02:45:16.123733 3084 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.65:33949
I20260806 02:45:16.125461 3084 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
I20260806 02:45:16.127841 3221 heartbeater.cc:347] Connected to a master server at 127.1.210.126:37725
I20260806 02:45:16.128093 3221 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:16.128800 3221 heartbeater.cc:511] Master 127.1.210.126:37725 requested a full tablet report, sending...
I20260806 02:45:16.131047 3011 ts_manager.cc:194] Registered new tserver with Master: e97a0e2cad6541d5a4c2a16f6ab5108d (127.1.210.65:33949)
I20260806 02:45:16.132575 3011 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.65@KRBTEST.COM'} at 127.1.210.65:52035
I20260806 02:45:16.135389 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 3084
I20260806 02:45:16.135573 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance
I20260806 02:45:16.136153 1865 external_mini_cluster.cc:1428] Setting key 11b67f90b5dc946c3cbd2977903b345e
WARNING: no policy specified for kudu/127.1.210.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab.
Entry for principal kudu/127.1.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab.
I20260806 02:45:16.184243 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.66:0
--local_ip_for_outbound_sockets=127.1.210.66
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:37725
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:42969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:16.525663 3225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:16.525974 3225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:16.526147 3225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:16.533712 3225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:16.533833 3225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:16.533975 3225 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:45:16.534073 3225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.66
I20260806 02:45:16.544621 3225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:42969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.210.126:37725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.66
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:16.546103 3225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:16.547669 3225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:16.561662 3230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:16.561751 3231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:16.561705 3233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:16.562916 3225 server_base.cc:1034] running on GCE node
I20260806 02:45:16.563613 3225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:16.564652 3225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:16.565878 3225 hybrid_clock.cc:648] HybridClock initialized: now 1785984316565787 us; error 65 us; skew 500 ppm
I20260806 02:45:16.566241 3225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:45:16 dist-test-slave-8tdl krb5kdc[2966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984316, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:16.569796 3225 init.cc:377] Logged in from keytab as kudu/127.1.210.66@KRBTEST.COM (short username kudu)
I20260806 02:45:16.571449 3225 webserver.cc:460] Webserver started at http://127.1.210.66:42707/ using document root <none> and password file <none>
I20260806 02:45:16.572307 3225 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:16.572402 3225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:16.572736 3225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:16.575902 3225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-1/data/instance:
uuid: "449d4e3b09e54611ac017a3319737734"
format_stamp: "Formatted at 2026-08-06 02:45:16 on dist-test-slave-8tdl"
server_key: "bad15ecb66c39c178775fade10baad4e"
server_key_iv: "8d84b8cfb1bef65feba5420b4569e2d1"
server_key_version: "encryptionkey@0"
I20260806 02:45:16.576751 3225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance:
uuid: "449d4e3b09e54611ac017a3319737734"
format_stamp: "Formatted at 2026-08-06 02:45:16 on dist-test-slave-8tdl"
server_key: "bad15ecb66c39c178775fade10baad4e"
server_key_iv: "8d84b8cfb1bef65feba5420b4569e2d1"
server_key_version: "encryptionkey@0"
I20260806 02:45:16.582952 3225 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 02:45:16.586836 3240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:16.588471 3225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:45:16.588673 3225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-1/wal
uuid: "449d4e3b09e54611ac017a3319737734"
format_stamp: "Formatted at 2026-08-06 02:45:16 on dist-test-slave-8tdl"
server_key: "bad15ecb66c39c178775fade10baad4e"
server_key_iv: "8d84b8cfb1bef65feba5420b4569e2d1"
server_key_version: "encryptionkey@0"
I20260806 02:45:16.588900 3225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:16.605362 3225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:16.610258 3225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:16.610556 3225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:16.613335 3225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:16.613459 3225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:16.613615 3225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:16.613689 3225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:45:16 dist-test-slave-8tdl krb5kdc[2966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984316, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:16.648927 3249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:16.615272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:37725 (local address 127.1.210.66:45419)
0806 02:45:16.616128 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:16.616165 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:16.618892 (+ 2727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:16.619866 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:16.619897 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:16.620830 (+ 933us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:16.621967 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:16.622001 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:16.623524 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:16.623529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:16.625784 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:16.625828 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:16.626788 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:16.626796 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:16.627124 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:16.628008 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:16.628035 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:16.632909 (+ 4874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:16.641749 (+ 8840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:16.641766 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:16.641804 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:16.642958 (+ 1154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:16.643794 (+ 836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:16.643802 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:16.643807 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:16.644166 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:16.645266 (+ 1100us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:16.645276 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:16.645817 (+ 541us) client_negotiation.cc:769] Sending connection context
0806 02:45:16.646215 (+ 398us) client_negotiation.cc:240] Negotiation successful
0806 02:45:16.647089 (+ 874us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":369,"threads_started":1}
I20260806 02:45:16.650453 3108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:16.614210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:45419 (local address 127.1.210.126:37725)
0806 02:45:16.614441 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:16.614447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:16.616318 (+ 1871us) server_negotiation.cc:407] Connection header received
0806 02:45:16.619281 (+ 2963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:16.619291 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:16.619411 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:16.619541 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:16.622193 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:16.623355 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:16.626010 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:16.626486 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:16.626645 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:16.636186 (+ 9541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:16.636195 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:16.636201 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:16.636274 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:16.641453 (+ 5179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:16.643173 (+ 1720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:16.643183 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:16.643187 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:16.643361 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:16.644353 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:16.644362 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:16.644365 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:16.644838 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:16.645171 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:16.646551 (+ 1380us) server_negotiation.cc:299] Negotiation successful
0806 02:45:16.650010 (+ 3459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 02:45:16.681246 3108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:16.662994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:58395 (local address 127.1.210.126:37725)
0806 02:45:16.663272 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:16.663281 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:16.663739 (+ 458us) server_negotiation.cc:407] Connection header received
0806 02:45:16.663958 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:16.663964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:16.664038 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:16.664148 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:16.665580 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:16.666533 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:16.669013 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:16.669619 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:16.669760 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:16.673027 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:16.673037 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:16.673044 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:16.673093 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:16.677171 (+ 4078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:16.678420 (+ 1249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:16.678427 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:16.678433 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:16.678607 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:16.679164 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:16.679172 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:16.679177 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:16.679717 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:16.679884 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:16.680639 (+ 755us) server_negotiation.cc:299] Negotiation successful
0806 02:45:16.680875 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260806 02:45:16.681332 3313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:16.662943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:37725 (local address 127.1.210.66:58395)
0806 02:45:16.663625 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:16.663652 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:16.663802 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:16.664179 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:16.664184 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:16.664928 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:16.665409 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:16.665426 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:16.666671 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:16.666704 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:16.668766 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:16.668789 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:16.670162 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:16.670173 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:16.670444 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:16.671282 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:16.671318 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:16.672724 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:16.677359 (+ 4635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:16.677365 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:16.677369 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:16.678220 (+ 851us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:16.678777 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:16.678784 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:16.678790 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:16.678993 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:16.679957 (+ 964us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:16.679962 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:16.680272 (+ 310us) client_negotiation.cc:769] Sending connection context
0806 02:45:16.680488 (+ 216us) client_negotiation.cc:240] Negotiation successful
0806 02:45:16.680875 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":376,"threads_started":1}
I20260806 02:45:16.693343 3363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.66:44591 every 8 connection(s)
I20260806 02:45:16.693583 3225 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.66:44591
I20260806 02:45:16.695026 3225 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
I20260806 02:45:16.696329 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 3225
I20260806 02:45:16.696576 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance
I20260806 02:45:16.697245 1865 external_mini_cluster.cc:1428] Setting key 90fb74e14ce9b63dad5fd0f43a908764
I20260806 02:45:16.699525 3364 heartbeater.cc:347] Connected to a master server at 127.1.210.126:37725
I20260806 02:45:16.709555 3364 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:16.710350 3364 heartbeater.cc:511] Master 127.1.210.126:37725 requested a full tablet report, sending...
I20260806 02:45:16.711974 3011 ts_manager.cc:194] Registered new tserver with Master: 449d4e3b09e54611ac017a3319737734 (127.1.210.66:44591)
I20260806 02:45:16.712819 3011 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.66@KRBTEST.COM'} at 127.1.210.66:45419
WARNING: no policy specified for kudu/127.1.210.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab.
Entry for principal kudu/127.1.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab.
I20260806 02:45:16.752046 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.67:0
--local_ip_for_outbound_sockets=127.1.210.67
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:37725
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:42969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:17.102514 3368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:17.102923 3368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:17.103120 3368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:17.111481 3368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:17.111660 3368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:17.111822 3368 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:45:17.111979 3368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.67
I20260806 02:45:17.124579 3368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:42969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.210.126:37725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.67
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:17.126384 3368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:17.128366 3368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:45:17.135679 3221 heartbeater.cc:503] Master 127.1.210.126:37725 was elected leader, sending a full tablet report...
W20260806 02:45:17.144352 3377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:17.144395 3374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:17.144423 3375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:17.145128 3368 server_base.cc:1034] running on GCE node
I20260806 02:45:17.145892 3368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:17.147068 3368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:17.148299 3368 hybrid_clock.cc:648] HybridClock initialized: now 1785984317148224 us; error 50 us; skew 500 ppm
I20260806 02:45:17.148717 3368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:45:17 dist-test-slave-8tdl krb5kdc[2966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984317, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:17.152292 3368 init.cc:377] Logged in from keytab as kudu/127.1.210.67@KRBTEST.COM (short username kudu)
I20260806 02:45:17.153975 3368 webserver.cc:460] Webserver started at http://127.1.210.67:37779/ using document root <none> and password file <none>
I20260806 02:45:17.154803 3368 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:17.154959 3368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:17.155303 3368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:17.158432 3368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-2/data/instance:
uuid: "64b51ce70efb49bf832abd4b15a5b900"
format_stamp: "Formatted at 2026-08-06 02:45:17 on dist-test-slave-8tdl"
server_key: "ed1770103babbc35153b3b8da0cd21d2"
server_key_iv: "a1ebf1a92baa0929fef9cf7ad0ffff9b"
server_key_version: "encryptionkey@0"
I20260806 02:45:17.159304 3368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance:
uuid: "64b51ce70efb49bf832abd4b15a5b900"
format_stamp: "Formatted at 2026-08-06 02:45:17 on dist-test-slave-8tdl"
server_key: "ed1770103babbc35153b3b8da0cd21d2"
server_key_iv: "a1ebf1a92baa0929fef9cf7ad0ffff9b"
server_key_version: "encryptionkey@0"
I20260806 02:45:17.164726 3368 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 02:45:17.168777 3384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:17.170128 3368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:45:17.170311 3368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-2/wal
uuid: "64b51ce70efb49bf832abd4b15a5b900"
format_stamp: "Formatted at 2026-08-06 02:45:17 on dist-test-slave-8tdl"
server_key: "ed1770103babbc35153b3b8da0cd21d2"
server_key_iv: "a1ebf1a92baa0929fef9cf7ad0ffff9b"
server_key_version: "encryptionkey@0"
I20260806 02:45:17.170532 3368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:17.211704 3368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:17.217281 3368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:17.217622 3368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:17.220676 3368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:17.220831 3368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:17.221011 3368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:17.221091 3368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:45:17 dist-test-slave-8tdl krb5kdc[2966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984317, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:17.250550 3394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:17.223004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:42759 (local address 127.1.210.126:37725)
0806 02:45:17.224473 (+ 1469us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:17.224482 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:17.224512 (+ 30us) server_negotiation.cc:407] Connection header received
0806 02:45:17.226412 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:17.226419 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:17.226521 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:17.226670 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:17.229670 (+ 3000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.231191 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.233991 (+ 2800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.234908 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.235099 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:17.240844 (+ 5745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:17.240857 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:17.240864 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:17.240934 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:17.245309 (+ 4375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:17.246807 (+ 1498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:17.246814 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:17.246818 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:17.247023 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:17.247896 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:17.247905 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:17.247911 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:17.248365 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:17.248587 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:17.250008 (+ 1421us) server_negotiation.cc:299] Negotiation successful
0806 02:45:17.250226 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1292,"thread_start_us":1169,"threads_started":1}
I20260806 02:45:17.253032 3393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:17.222448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:37725 (local address 127.1.210.67:42759)
0806 02:45:17.223653 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:17.223731 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:17.226058 (+ 2327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:17.227047 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:17.227057 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:17.228257 (+ 1200us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:17.229297 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:17.229344 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.231402 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.231411 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:17.233743 (+ 2332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:17.233757 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.235276 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.235297 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:17.235702 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:17.236699 (+ 997us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:17.236725 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:17.240499 (+ 3774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:17.245497 (+ 4998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:17.245513 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:17.245542 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:17.246529 (+ 987us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:17.247235 (+ 706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:17.247242 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:17.247246 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:17.247632 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:17.248598 (+ 966us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:17.248615 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:17.249362 (+ 747us) client_negotiation.cc:769] Sending connection context
0806 02:45:17.249844 (+ 482us) client_negotiation.cc:240] Negotiation successful
0806 02:45:17.250727 (+ 883us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"thread_start_us":458,"threads_started":1}
I20260806 02:45:17.274323 3394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:17.259397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:53295 (local address 127.1.210.126:37725)
0806 02:45:17.259625 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:17.259635 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:17.260557 (+ 922us) server_negotiation.cc:407] Connection header received
0806 02:45:17.260779 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:17.260788 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:17.260874 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:17.260988 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:17.262093 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.262951 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.264534 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.264997 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.265157 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:17.267353 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:17.267358 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:17.267362 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:17.267397 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:17.270974 (+ 3577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:17.271984 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:17.271993 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:17.272000 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:17.272196 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:17.272681 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:17.272690 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:17.272693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:17.273008 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:17.273198 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:17.273614 (+ 416us) server_negotiation.cc:299] Negotiation successful
0806 02:45:17.273878 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 02:45:17.274482 3456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:17.259429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:37725 (local address 127.1.210.67:53295)
0806 02:45:17.260442 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:17.260460 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:17.260642 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:17.261016 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:17.261021 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:17.261516 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:17.261831 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:17.261841 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.263117 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.263123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:17.264357 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:17.264370 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.265207 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.265215 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:17.265402 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:17.266107 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:17.266126 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:17.267114 (+ 988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:17.271148 (+ 4034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:17.271154 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:17.271159 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:17.271832 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:17.272358 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:17.272366 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:17.272373 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:17.272535 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:17.273140 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:17.273145 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:17.273351 (+ 206us) client_negotiation.cc:769] Sending connection context
0806 02:45:17.273510 (+ 159us) client_negotiation.cc:240] Negotiation successful
0806 02:45:17.273964 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"thread_start_us":696,"threads_started":1}
I20260806 02:45:17.289951 3368 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.67:45229
I20260806 02:45:17.289963 3504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.67:45229 every 8 connection(s)
I20260806 02:45:17.291478 3368 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
I20260806 02:45:17.294193 3506 heartbeater.cc:347] Connected to a master server at 127.1.210.126:37725
I20260806 02:45:17.294466 3506 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:17.294458 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 3368
I20260806 02:45:17.294669 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance
I20260806 02:45:17.295328 1865 external_mini_cluster.cc:1428] Setting key c73d5a3a1181961f3f1111a78ae70bf8
I20260806 02:45:17.295367 3506 heartbeater.cc:511] Master 127.1.210.126:37725 requested a full tablet report, sending...
I20260806 02:45:17.297026 3011 ts_manager.cc:194] Registered new tserver with Master: 64b51ce70efb49bf832abd4b15a5b900 (127.1.210.67:45229)
I20260806 02:45:17.298224 3011 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.67@KRBTEST.COM'} at 127.1.210.67:42759
I20260806 02:45:17.298849 1865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:45:17 dist-test-slave-8tdl krb5kdc[2966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984317, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 02:45:17 dist-test-slave-8tdl krb5kdc[2966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984317, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:17.352165 3394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:17.337265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60086 (local address 127.1.210.126:37725)
0806 02:45:17.337508 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:17.337514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:17.337849 (+ 335us) server_negotiation.cc:407] Connection header received
0806 02:45:17.338101 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:17.338106 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:17.338173 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:17.338286 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:17.339852 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.340583 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.341901 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.342315 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.342462 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:17.346853 (+ 4391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:17.346859 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:17.346862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:17.346904 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:17.349492 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:17.350207 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:17.350212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:17.350214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:17.350348 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:17.350855 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:17.350859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:17.350861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:17.351163 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:17.351385 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:17.351628 (+ 243us) server_negotiation.cc:299] Negotiation successful
0806 02:45:17.351794 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260806 02:45:17.357110 3011 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:45:17.364045 3011 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:45:17.404027 3519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:17.376609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:33949 (local address 127.0.0.1:42070)
0806 02:45:17.391798 (+ 15189us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:17.391862 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:17.392081 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:17.392952 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:17.392959 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:17.392992 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:17.393403 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:17.393421 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.396255 (+ 2834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.396263 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:17.399590 (+ 3327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:17.399613 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.402843 (+ 3230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.402849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:17.403150 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:17.403256 (+ 106us) client_negotiation.cc:769] Sending connection context
0806 02:45:17.403359 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 02:45:17.403551 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14867,"mutex_wait_us":3,"thread_start_us":15211,"threads_started":1}
I20260806 02:45:17.404212 3521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:17.376696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:45229 (local address 127.0.0.1:53888)
0806 02:45:17.392378 (+ 15682us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:17.392408 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:17.392556 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:17.394620 (+ 2064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:17.394628 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:17.394664 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:17.395258 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:17.395273 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.397681 (+ 2408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.397689 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:17.400763 (+ 3074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:17.400775 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.403151 (+ 2376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.403157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:17.403527 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:17.403562 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 02:45:17.403637 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 02:45:17.403773 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15528,"mutex_wait_us":116,"thread_start_us":15967,"threads_started":1}
I20260806 02:45:17.404886 3520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:17.376777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42070 (local address 127.1.210.65:33949)
0806 02:45:17.391892 (+ 15115us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:17.391902 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:17.392306 (+ 404us) server_negotiation.cc:407] Connection header received
0806 02:45:17.392415 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:17.392423 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:17.392693 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:17.393661 (+ 968us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:17.393929 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.396057 (+ 2128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.399867 (+ 3810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.402080 (+ 2213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.403529 (+ 1449us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:17.403752 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:17.403834 (+ 82us) server_negotiation.cc:299] Negotiation successful
0806 02:45:17.404020 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14937,"thread_start_us":14780,"threads_started":1}
I20260806 02:45:17.405028 3522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:17.377354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53888 (local address 127.1.210.67:45229)
0806 02:45:17.393493 (+ 16139us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:17.393502 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:17.393535 (+ 33us) server_negotiation.cc:407] Connection header received
0806 02:45:17.393633 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:17.393644 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:17.394023 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:17.394253 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:17.395515 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.397452 (+ 1937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.400952 (+ 3500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.402913 (+ 1961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.403069 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:17.403399 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:17.404465 (+ 1066us) server_negotiation.cc:299] Negotiation successful
0806 02:45:17.404682 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15968,"thread_start_us":15795,"threads_started":1}
I20260806 02:45:17.408838 3150 tablet_service.cc:1467] Processing CreateTablet for tablet b0cfd2a7839e4f549e39a11c29f46199 (DEFAULT_TABLE table=test-table [id=4932ddb9ad3c440699fd5034773c7bd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:17.408845 3435 tablet_service.cc:1467] Processing CreateTablet for tablet b0cfd2a7839e4f549e39a11c29f46199 (DEFAULT_TABLE table=test-table [id=4932ddb9ad3c440699fd5034773c7bd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:17.410123 3150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0cfd2a7839e4f549e39a11c29f46199. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:17.410266 3435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0cfd2a7839e4f549e39a11c29f46199. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:17.411237 3524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:17.393545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:44591 (local address 127.0.0.1:59744)
0806 02:45:17.403063 (+ 9518us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:17.403091 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:17.403225 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:17.404267 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:17.404337 (+ 70us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:17.404365 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:17.404767 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:17.404779 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.406338 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.406343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:17.408441 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:17.408458 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.410454 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.410462 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:17.410628 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:17.410661 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 02:45:17.410784 (+ 123us) client_negotiation.cc:240] Negotiation successful
0806 02:45:17.410912 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9366,"thread_start_us":9208,"threads_started":1}
I20260806 02:45:17.411280 3523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:17.392984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59744 (local address 127.1.210.66:44591)
0806 02:45:17.402630 (+ 9646us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:17.402643 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:17.403233 (+ 590us) server_negotiation.cc:407] Connection header received
0806 02:45:17.403459 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:17.403469 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:17.403822 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:17.404038 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:17.405098 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.406168 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.408617 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.410140 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.410289 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:17.410487 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:17.410828 (+ 341us) server_negotiation.cc:299] Negotiation successful
0806 02:45:17.411000 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9485,"thread_start_us":9381,"threads_started":1}
I20260806 02:45:17.415431 3293 tablet_service.cc:1467] Processing CreateTablet for tablet b0cfd2a7839e4f549e39a11c29f46199 (DEFAULT_TABLE table=test-table [id=4932ddb9ad3c440699fd5034773c7bd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:17.416898 3293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0cfd2a7839e4f549e39a11c29f46199. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:17.425282 3526 tablet_bootstrap.cc:492] T b0cfd2a7839e4f549e39a11c29f46199 P 64b51ce70efb49bf832abd4b15a5b900: Bootstrap starting.
I20260806 02:45:17.427671 3525 tablet_bootstrap.cc:492] T b0cfd2a7839e4f549e39a11c29f46199 P e97a0e2cad6541d5a4c2a16f6ab5108d: Bootstrap starting.
I20260806 02:45:17.430451 3526 tablet_bootstrap.cc:654] T b0cfd2a7839e4f549e39a11c29f46199 P 64b51ce70efb49bf832abd4b15a5b900: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:17.432202 3526 log.cc:824] T b0cfd2a7839e4f549e39a11c29f46199 P 64b51ce70efb49bf832abd4b15a5b900: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:17.432456 3525 tablet_bootstrap.cc:654] T b0cfd2a7839e4f549e39a11c29f46199 P e97a0e2cad6541d5a4c2a16f6ab5108d: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:17.435081 3525 log.cc:824] T b0cfd2a7839e4f549e39a11c29f46199 P e97a0e2cad6541d5a4c2a16f6ab5108d: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:17.436045 3528 tablet_bootstrap.cc:492] T b0cfd2a7839e4f549e39a11c29f46199 P 449d4e3b09e54611ac017a3319737734: Bootstrap starting.
I20260806 02:45:17.436519 3526 tablet_bootstrap.cc:492] T b0cfd2a7839e4f549e39a11c29f46199 P 64b51ce70efb49bf832abd4b15a5b900: No bootstrap required, opened a new log
I20260806 02:45:17.436903 3526 ts_tablet_manager.cc:1397] T b0cfd2a7839e4f549e39a11c29f46199 P 64b51ce70efb49bf832abd4b15a5b900: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260806 02:45:17.441191 3528 tablet_bootstrap.cc:654] T b0cfd2a7839e4f549e39a11c29f46199 P 449d4e3b09e54611ac017a3319737734: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:17.443033 3528 log.cc:824] T b0cfd2a7839e4f549e39a11c29f46199 P 449d4e3b09e54611ac017a3319737734: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:17.447618 3525 tablet_bootstrap.cc:492] T b0cfd2a7839e4f549e39a11c29f46199 P e97a0e2cad6541d5a4c2a16f6ab5108d: No bootstrap required, opened a new log
I20260806 02:45:17.447688 3528 tablet_bootstrap.cc:492] T b0cfd2a7839e4f549e39a11c29f46199 P 449d4e3b09e54611ac017a3319737734: No bootstrap required, opened a new log
I20260806 02:45:17.447969 3525 ts_tablet_manager.cc:1397] T b0cfd2a7839e4f549e39a11c29f46199 P e97a0e2cad6541d5a4c2a16f6ab5108d: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.009s
I20260806 02:45:17.448169 3528 ts_tablet_manager.cc:1397] T b0cfd2a7839e4f549e39a11c29f46199 P 449d4e3b09e54611ac017a3319737734: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.008s
I20260806 02:45:17.452049 3526 raft_consensus.cc:348] T b0cfd2a7839e4f549e39a11c29f46199 P 64b51ce70efb49bf832abd4b15a5b900 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } }
I20260806 02:45:17.452579 3526 raft_consensus.cc:374] T b0cfd2a7839e4f549e39a11c29f46199 P 64b51ce70efb49bf832abd4b15a5b900 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:17.452682 3526 raft_consensus.cc:729] T b0cfd2a7839e4f549e39a11c29f46199 P 64b51ce70efb49bf832abd4b15a5b900 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64b51ce70efb49bf832abd4b15a5b900, State: Initialized, Role: FOLLOWER
I20260806 02:45:17.453383 3526 consensus_queue.cc:260] T b0cfd2a7839e4f549e39a11c29f46199 P 64b51ce70efb49bf832abd4b15a5b900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } }
I20260806 02:45:17.460052 3506 heartbeater.cc:503] Master 127.1.210.126:37725 was elected leader, sending a full tablet report...
I20260806 02:45:17.461056 3526 ts_tablet_manager.cc:1428] T b0cfd2a7839e4f549e39a11c29f46199 P 64b51ce70efb49bf832abd4b15a5b900: Time spent starting tablet: real 0.024s user 0.013s sys 0.011s
I20260806 02:45:17.461936 3528 raft_consensus.cc:348] T b0cfd2a7839e4f549e39a11c29f46199 P 449d4e3b09e54611ac017a3319737734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } }
I20260806 02:45:17.462913 3528 raft_consensus.cc:374] T b0cfd2a7839e4f549e39a11c29f46199 P 449d4e3b09e54611ac017a3319737734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:17.463177 3528 raft_consensus.cc:729] T b0cfd2a7839e4f549e39a11c29f46199 P 449d4e3b09e54611ac017a3319737734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 449d4e3b09e54611ac017a3319737734, State: Initialized, Role: FOLLOWER
I20260806 02:45:17.464164 3528 consensus_queue.cc:260] T b0cfd2a7839e4f549e39a11c29f46199 P 449d4e3b09e54611ac017a3319737734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } }
I20260806 02:45:17.464927 3525 raft_consensus.cc:348] T b0cfd2a7839e4f549e39a11c29f46199 P e97a0e2cad6541d5a4c2a16f6ab5108d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } }
I20260806 02:45:17.465507 3525 raft_consensus.cc:374] T b0cfd2a7839e4f549e39a11c29f46199 P e97a0e2cad6541d5a4c2a16f6ab5108d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:17.465677 3525 raft_consensus.cc:729] T b0cfd2a7839e4f549e39a11c29f46199 P e97a0e2cad6541d5a4c2a16f6ab5108d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e97a0e2cad6541d5a4c2a16f6ab5108d, State: Initialized, Role: FOLLOWER
I20260806 02:45:17.466424 3525 consensus_queue.cc:260] T b0cfd2a7839e4f549e39a11c29f46199 P e97a0e2cad6541d5a4c2a16f6ab5108d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } }
I20260806 02:45:17.472463 3525 ts_tablet_manager.cc:1428] T b0cfd2a7839e4f549e39a11c29f46199 P e97a0e2cad6541d5a4c2a16f6ab5108d: Time spent starting tablet: real 0.024s user 0.013s sys 0.011s
I20260806 02:45:17.473860 3364 heartbeater.cc:503] Master 127.1.210.126:37725 was elected leader, sending a full tablet report...
I20260806 02:45:17.474426 3528 ts_tablet_manager.cc:1428] T b0cfd2a7839e4f549e39a11c29f46199 P 449d4e3b09e54611ac017a3319737734: Time spent starting tablet: real 0.026s user 0.013s sys 0.011s
W20260806 02:45:17.543609 3507 tablet.cc:2369] T b0cfd2a7839e4f549e39a11c29f46199 P 64b51ce70efb49bf832abd4b15a5b900: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:45:17.628677 3222 tablet.cc:2369] T b0cfd2a7839e4f549e39a11c29f46199 P e97a0e2cad6541d5a4c2a16f6ab5108d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:45:17.697227 3365 tablet.cc:2369] T b0cfd2a7839e4f549e39a11c29f46199 P 449d4e3b09e54611ac017a3319737734: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:45:17.767074 3532 raft_consensus.cc:482] T b0cfd2a7839e4f549e39a11c29f46199 P 449d4e3b09e54611ac017a3319737734 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:45:17.767428 3532 raft_consensus.cc:504] T b0cfd2a7839e4f549e39a11c29f46199 P 449d4e3b09e54611ac017a3319737734 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } }
I20260806 02:45:17.769766 3532 leader_election.cc:288] T b0cfd2a7839e4f549e39a11c29f46199 P 449d4e3b09e54611ac017a3319737734 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e97a0e2cad6541d5a4c2a16f6ab5108d (127.1.210.65:33949), 64b51ce70efb49bf832abd4b15a5b900 (127.1.210.67:45229)
I20260806 02:45:17.775019 3533 raft_consensus.cc:482] T b0cfd2a7839e4f549e39a11c29f46199 P e97a0e2cad6541d5a4c2a16f6ab5108d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:45:17.775620 3533 raft_consensus.cc:504] T b0cfd2a7839e4f549e39a11c29f46199 P e97a0e2cad6541d5a4c2a16f6ab5108d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } }
I20260806 02:45:17.778954 3533 leader_election.cc:288] T b0cfd2a7839e4f549e39a11c29f46199 P e97a0e2cad6541d5a4c2a16f6ab5108d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64b51ce70efb49bf832abd4b15a5b900 (127.1.210.67:45229), 449d4e3b09e54611ac017a3319737734 (127.1.210.66:44591)
I20260806 02:45:17.781708 3522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:17.770278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:44509 (local address 127.1.210.67:45229)
0806 02:45:17.770517 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:17.770527 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:17.771423 (+ 896us) server_negotiation.cc:407] Connection header received
0806 02:45:17.771636 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:17.771640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:17.771703 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:17.771798 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:17.772842 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.773904 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.777085 (+ 3181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.778545 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.778801 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:17.778890 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:17.781301 (+ 2411us) server_negotiation.cc:299] Negotiation successful
0806 02:45:17.781438 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 02:45:17.781826 3535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:17.770136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:45229 (local address 127.1.210.66:44509)
0806 02:45:17.771286 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:17.771319 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:17.771461 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:17.771822 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:17.771827 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:17.771852 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:17.772627 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:17.772641 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.774081 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.774089 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:17.776911 (+ 2822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:17.776926 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.780800 (+ 3874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.780814 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:17.781103 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:17.781151 (+ 48us) client_negotiation.cc:769] Sending connection context
0806 02:45:17.781267 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 02:45:17.781412 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1029,"mutex_wait_us":109,"thread_start_us":836,"threads_started":1}
I20260806 02:45:17.783195 3520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:17.770265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:46757 (local address 127.1.210.65:33949)
0806 02:45:17.770517 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:17.770527 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:17.771303 (+ 776us) server_negotiation.cc:407] Connection header received
0806 02:45:17.771632 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:17.771638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:17.771710 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:17.771814 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:17.772947 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.773986 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.777528 (+ 3542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.780382 (+ 2854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.780638 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:17.780754 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:17.782705 (+ 1951us) server_negotiation.cc:299] Negotiation successful
0806 02:45:17.782857 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:45:17.783141 3460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0cfd2a7839e4f549e39a11c29f46199" candidate_uuid: "449d4e3b09e54611ac017a3319737734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64b51ce70efb49bf832abd4b15a5b900" is_pre_election: true
I20260806 02:45:17.783720 3460 raft_consensus.cc:2454] T b0cfd2a7839e4f549e39a11c29f46199 P 64b51ce70efb49bf832abd4b15a5b900 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 449d4e3b09e54611ac017a3319737734 in term 0.
I20260806 02:45:17.784314 3534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:17.770039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:33949 (local address 127.1.210.66:46757)
0806 02:45:17.771156 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:17.771182 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:17.771413 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:17.771855 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:17.771859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:17.771882 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:17.772722 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:17.772735 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.774135 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.774139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:17.777311 (+ 3172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:17.777326 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.782263 (+ 4937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.782273 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:17.782488 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:17.782522 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 02:45:17.783431 (+ 909us) client_negotiation.cc:240] Negotiation successful
0806 02:45:17.783576 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"mutex_wait_us":93,"thread_start_us":548,"threads_started":1}
I20260806 02:45:17.785670 3242 leader_election.cc:302] T b0cfd2a7839e4f549e39a11c29f46199 P 449d4e3b09e54611ac017a3319737734 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 449d4e3b09e54611ac017a3319737734, 64b51ce70efb49bf832abd4b15a5b900; no voters:
I20260806 02:45:17.786273 3532 raft_consensus.cc:2781] T b0cfd2a7839e4f549e39a11c29f46199 P 449d4e3b09e54611ac017a3319737734 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:45:17.786212 3171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0cfd2a7839e4f549e39a11c29f46199" candidate_uuid: "449d4e3b09e54611ac017a3319737734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" is_pre_election: true
I20260806 02:45:17.786536 3532 raft_consensus.cc:482] T b0cfd2a7839e4f549e39a11c29f46199 P 449d4e3b09e54611ac017a3319737734 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:45:17.786808 3532 raft_consensus.cc:3037] T b0cfd2a7839e4f549e39a11c29f46199 P 449d4e3b09e54611ac017a3319737734 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:17.786870 3171 raft_consensus.cc:2454] T b0cfd2a7839e4f549e39a11c29f46199 P e97a0e2cad6541d5a4c2a16f6ab5108d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 449d4e3b09e54611ac017a3319737734 in term 0.
I20260806 02:45:17.789803 3532 raft_consensus.cc:504] T b0cfd2a7839e4f549e39a11c29f46199 P 449d4e3b09e54611ac017a3319737734 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } }
I20260806 02:45:17.791231 3460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0cfd2a7839e4f549e39a11c29f46199" candidate_uuid: "449d4e3b09e54611ac017a3319737734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64b51ce70efb49bf832abd4b15a5b900"
I20260806 02:45:17.791296 3171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0cfd2a7839e4f549e39a11c29f46199" candidate_uuid: "449d4e3b09e54611ac017a3319737734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d"
I20260806 02:45:17.791484 3460 raft_consensus.cc:3037] T b0cfd2a7839e4f549e39a11c29f46199 P 64b51ce70efb49bf832abd4b15a5b900 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:17.791595 3171 raft_consensus.cc:3037] T b0cfd2a7839e4f549e39a11c29f46199 P e97a0e2cad6541d5a4c2a16f6ab5108d [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:17.793195 3532 leader_election.cc:288] T b0cfd2a7839e4f549e39a11c29f46199 P 449d4e3b09e54611ac017a3319737734 [CANDIDATE]: Term 1 election: Requested vote from peers e97a0e2cad6541d5a4c2a16f6ab5108d (127.1.210.65:33949), 64b51ce70efb49bf832abd4b15a5b900 (127.1.210.67:45229)
I20260806 02:45:17.796205 3171 raft_consensus.cc:2454] T b0cfd2a7839e4f549e39a11c29f46199 P e97a0e2cad6541d5a4c2a16f6ab5108d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 449d4e3b09e54611ac017a3319737734 in term 1.
I20260806 02:45:17.796206 3460 raft_consensus.cc:2454] T b0cfd2a7839e4f549e39a11c29f46199 P 64b51ce70efb49bf832abd4b15a5b900 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 449d4e3b09e54611ac017a3319737734 in term 1.
I20260806 02:45:17.796675 3244 leader_election.cc:302] T b0cfd2a7839e4f549e39a11c29f46199 P 449d4e3b09e54611ac017a3319737734 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 449d4e3b09e54611ac017a3319737734, e97a0e2cad6541d5a4c2a16f6ab5108d; no voters:
I20260806 02:45:17.797224 3532 raft_consensus.cc:2781] T b0cfd2a7839e4f549e39a11c29f46199 P 449d4e3b09e54611ac017a3319737734 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:17.799279 3532 raft_consensus.cc:686] T b0cfd2a7839e4f549e39a11c29f46199 P 449d4e3b09e54611ac017a3319737734 [term 1 LEADER]: Becoming Leader. State: Replica: 449d4e3b09e54611ac017a3319737734, State: Running, Role: LEADER
I20260806 02:45:17.800024 3537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:17.778652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:55015 (local address 127.1.210.67:45229)
0806 02:45:17.779580 (+ 928us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:17.779587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:17.790300 (+ 10713us) server_negotiation.cc:407] Connection header received
0806 02:45:17.790781 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:17.790798 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:17.790972 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:17.791133 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:17.792964 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.794595 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.797326 (+ 2731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.798305 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.798555 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:17.798651 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:17.799357 (+ 706us) server_negotiation.cc:299] Negotiation successful
0806 02:45:17.799535 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":786,"thread_start_us":1413,"threads_started":1}
I20260806 02:45:17.800040 3532 consensus_queue.cc:237] T b0cfd2a7839e4f549e39a11c29f46199 P 449d4e3b09e54611ac017a3319737734 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } }
I20260806 02:45:17.800858 3523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:17.778489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:53285 (local address 127.1.210.66:44591)
0806 02:45:17.778755 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:17.778773 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:17.781612 (+ 2839us) server_negotiation.cc:407] Connection header received
0806 02:45:17.781949 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:17.781958 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:17.782061 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:17.782208 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:17.783962 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.785523 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.796874 (+ 11351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.799255 (+ 2381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.799452 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:17.799520 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:17.800351 (+ 831us) server_negotiation.cc:299] Negotiation successful
0806 02:45:17.800493 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:45:17.801055 3536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:17.779729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:45229 (local address 127.1.210.65:55015)
0806 02:45:17.790114 (+ 10385us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:17.790173 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:17.790385 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:17.791449 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:17.791454 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:17.791480 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:17.792674 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:17.792690 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.795020 (+ 2330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.795024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:17.797088 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:17.797103 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.798619 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.798627 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:17.799074 (+ 447us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:17.799130 (+ 56us) client_negotiation.cc:769] Sending connection context
0806 02:45:17.800348 (+ 1218us) client_negotiation.cc:240] Negotiation successful
0806 02:45:17.800506 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10219,"thread_start_us":1037,"threads_started":1}
I20260806 02:45:17.801836 3538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:17.778322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:44591 (local address 127.1.210.65:53285)
0806 02:45:17.781465 (+ 3143us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:17.781501 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:17.781756 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:17.782556 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:17.782564 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:17.782597 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:17.783741 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:17.783758 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.787474 (+ 3716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.787482 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:17.796676 (+ 9194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:17.796688 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.799864 (+ 3176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.799872 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:17.800114 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:17.800155 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 02:45:17.801104 (+ 949us) client_negotiation.cc:240] Negotiation successful
0806 02:45:17.801260 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2923,"thread_start_us":12154,"threads_started":1}
I20260806 02:45:17.802096 3460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0cfd2a7839e4f549e39a11c29f46199" candidate_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64b51ce70efb49bf832abd4b15a5b900" is_pre_election: true
I20260806 02:45:17.802932 3318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0cfd2a7839e4f549e39a11c29f46199" candidate_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449d4e3b09e54611ac017a3319737734" is_pre_election: true
I20260806 02:45:17.804273 3460 raft_consensus.cc:2379] T b0cfd2a7839e4f549e39a11c29f46199 P 64b51ce70efb49bf832abd4b15a5b900 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e97a0e2cad6541d5a4c2a16f6ab5108d in current term 1: Already voted for candidate 449d4e3b09e54611ac017a3319737734 in this term.
I20260806 02:45:17.806588 3101 leader_election.cc:302] T b0cfd2a7839e4f549e39a11c29f46199 P e97a0e2cad6541d5a4c2a16f6ab5108d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e97a0e2cad6541d5a4c2a16f6ab5108d; no voters: 449d4e3b09e54611ac017a3319737734, 64b51ce70efb49bf832abd4b15a5b900
I20260806 02:45:17.807278 3533 raft_consensus.cc:2726] T b0cfd2a7839e4f549e39a11c29f46199 P e97a0e2cad6541d5a4c2a16f6ab5108d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 02:45:17.807958 3010 catalog_manager.cc:5495] T b0cfd2a7839e4f549e39a11c29f46199 P 449d4e3b09e54611ac017a3319737734 reported cstate change: term changed from 0 to 1, leader changed from <none> to 449d4e3b09e54611ac017a3319737734 (127.1.210.66). New cstate: current_term: 1 leader_uuid: "449d4e3b09e54611ac017a3319737734" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } health_report { overall_health: HEALTHY } } }
I20260806 02:45:17.871798 3523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:17.863216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59754 (local address 127.1.210.66:44591)
0806 02:45:17.863393 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:17.863398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:17.864233 (+ 835us) server_negotiation.cc:407] Connection header received
0806 02:45:17.864536 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:17.864540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:17.864623 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:17.864753 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:17.865568 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.866631 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.868692 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:17.869337 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:17.869446 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:17.870291 (+ 845us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:17.870996 (+ 705us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:17.871153 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:17.871338 (+ 185us) server_negotiation.cc:299] Negotiation successful
0806 02:45:17.871413 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:45:17.883137 3171 raft_consensus.cc:1261] T b0cfd2a7839e4f549e39a11c29f46199 P e97a0e2cad6541d5a4c2a16f6ab5108d [term 1 FOLLOWER]: Refusing update from remote peer 449d4e3b09e54611ac017a3319737734: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:17.883363 3460 raft_consensus.cc:1261] T b0cfd2a7839e4f549e39a11c29f46199 P 64b51ce70efb49bf832abd4b15a5b900 [term 1 FOLLOWER]: Refusing update from remote peer 449d4e3b09e54611ac017a3319737734: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:17.884276 3539 consensus_queue.cc:1035] T b0cfd2a7839e4f549e39a11c29f46199 P 449d4e3b09e54611ac017a3319737734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:17.884824 3532 consensus_queue.cc:1035] T b0cfd2a7839e4f549e39a11c29f46199 P 449d4e3b09e54611ac017a3319737734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:17.897158 3543 mvcc.cc:205] Tried to move back new op lower bound from 7315391766036443136 to 7315391765717721088. Current Snapshot: MvccSnapshot[applied={T|T < 7315391766036443136}]
I20260806 02:45:17.906718 3546 mvcc.cc:205] Tried to move back new op lower bound from 7315391766036443136 to 7315391765717721088. Current Snapshot: MvccSnapshot[applied={T|T < 7315391766036443136}]
I20260806 02:45:17.923314 3011 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:17.923666 3011 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:17.927632 3011 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 69e101b4d0f249a4b5cf7a908e31be9c: Sending DeleteTablet for 3 replicas of tablet b0cfd2a7839e4f549e39a11c29f46199
I20260806 02:45:17.929219 3435 tablet_service.cc:1514] Processing DeleteTablet for tablet b0cfd2a7839e4f549e39a11c29f46199 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:17 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:53888
I20260806 02:45:17.929275 3150 tablet_service.cc:1514] Processing DeleteTablet for tablet b0cfd2a7839e4f549e39a11c29f46199 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:17 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:42070
I20260806 02:45:17.929842 3293 tablet_service.cc:1514] Processing DeleteTablet for tablet b0cfd2a7839e4f549e39a11c29f46199 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:17 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:59744
I20260806 02:45:17.932226 3011 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:45:17.933248 3011 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:45:17.945611 3434 tablet_service.cc:1467] Processing CreateTablet for tablet feb1e52541414a82b49c5cc67ca48b76 (DEFAULT_TABLE table=test-table [id=87d9ec117dd74ac3b657470ed5ebd880]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:17.945915 3291 tablet_service.cc:1467] Processing CreateTablet for tablet feb1e52541414a82b49c5cc67ca48b76 (DEFAULT_TABLE table=test-table [id=87d9ec117dd74ac3b657470ed5ebd880]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:17.946270 3434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feb1e52541414a82b49c5cc67ca48b76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:17.946544 3291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feb1e52541414a82b49c5cc67ca48b76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:17.947419 3149 tablet_service.cc:1467] Processing CreateTablet for tablet feb1e52541414a82b49c5cc67ca48b76 (DEFAULT_TABLE table=test-table [id=87d9ec117dd74ac3b657470ed5ebd880]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:17.948225 3149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feb1e52541414a82b49c5cc67ca48b76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:17.953733 3554 tablet_replica.cc:330] T b0cfd2a7839e4f549e39a11c29f46199 P 449d4e3b09e54611ac017a3319737734: stopping tablet replica
I20260806 02:45:17.954478 3554 raft_consensus.cc:2229] T b0cfd2a7839e4f549e39a11c29f46199 P 449d4e3b09e54611ac017a3319737734 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:45:17.954845 3553 tablet_replica.cc:330] T b0cfd2a7839e4f549e39a11c29f46199 P 64b51ce70efb49bf832abd4b15a5b900: stopping tablet replica
I20260806 02:45:17.955471 3553 raft_consensus.cc:2229] T b0cfd2a7839e4f549e39a11c29f46199 P 64b51ce70efb49bf832abd4b15a5b900 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:17.956352 3526 tablet_bootstrap.cc:492] T feb1e52541414a82b49c5cc67ca48b76 P 64b51ce70efb49bf832abd4b15a5b900: Bootstrap starting.
I20260806 02:45:17.957415 3554 raft_consensus.cc:2258] T b0cfd2a7839e4f549e39a11c29f46199 P 449d4e3b09e54611ac017a3319737734 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:17.957716 3553 raft_consensus.cc:2258] T b0cfd2a7839e4f549e39a11c29f46199 P 64b51ce70efb49bf832abd4b15a5b900 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:17.958771 3526 tablet_bootstrap.cc:654] T feb1e52541414a82b49c5cc67ca48b76 P 64b51ce70efb49bf832abd4b15a5b900: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:17.961695 3553 ts_tablet_manager.cc:1905] T b0cfd2a7839e4f549e39a11c29f46199 P 64b51ce70efb49bf832abd4b15a5b900: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:17.961874 3555 tablet_replica.cc:330] T b0cfd2a7839e4f549e39a11c29f46199 P e97a0e2cad6541d5a4c2a16f6ab5108d: stopping tablet replica
I20260806 02:45:17.964927 3525 tablet_bootstrap.cc:492] T feb1e52541414a82b49c5cc67ca48b76 P e97a0e2cad6541d5a4c2a16f6ab5108d: Bootstrap starting.
I20260806 02:45:17.965234 3528 tablet_bootstrap.cc:492] T feb1e52541414a82b49c5cc67ca48b76 P 449d4e3b09e54611ac017a3319737734: Bootstrap starting.
I20260806 02:45:17.967922 3525 tablet_bootstrap.cc:654] T feb1e52541414a82b49c5cc67ca48b76 P e97a0e2cad6541d5a4c2a16f6ab5108d: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:17.968954 3528 tablet_bootstrap.cc:654] T feb1e52541414a82b49c5cc67ca48b76 P 449d4e3b09e54611ac017a3319737734: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:17.972064 3528 tablet_bootstrap.cc:492] T feb1e52541414a82b49c5cc67ca48b76 P 449d4e3b09e54611ac017a3319737734: No bootstrap required, opened a new log
I20260806 02:45:17.972151 3526 tablet_bootstrap.cc:492] T feb1e52541414a82b49c5cc67ca48b76 P 64b51ce70efb49bf832abd4b15a5b900: No bootstrap required, opened a new log
I20260806 02:45:17.972321 3528 ts_tablet_manager.cc:1397] T feb1e52541414a82b49c5cc67ca48b76 P 449d4e3b09e54611ac017a3319737734: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260806 02:45:17.972558 3526 ts_tablet_manager.cc:1397] T feb1e52541414a82b49c5cc67ca48b76 P 64b51ce70efb49bf832abd4b15a5b900: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.007s
I20260806 02:45:17.972651 3554 ts_tablet_manager.cc:1905] T b0cfd2a7839e4f549e39a11c29f46199 P 449d4e3b09e54611ac017a3319737734: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:17.973675 3553 ts_tablet_manager.cc:1918] T b0cfd2a7839e4f549e39a11c29f46199 P 64b51ce70efb49bf832abd4b15a5b900: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:45:17.974001 3553 log.cc:1196] T b0cfd2a7839e4f549e39a11c29f46199 P 64b51ce70efb49bf832abd4b15a5b900: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-2/wal/wals/b0cfd2a7839e4f549e39a11c29f46199
I20260806 02:45:17.974190 3555 raft_consensus.cc:2229] T b0cfd2a7839e4f549e39a11c29f46199 P e97a0e2cad6541d5a4c2a16f6ab5108d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:17.973459 3528 raft_consensus.cc:348] T feb1e52541414a82b49c5cc67ca48b76 P 449d4e3b09e54611ac017a3319737734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } }
I20260806 02:45:17.974547 3528 raft_consensus.cc:374] T feb1e52541414a82b49c5cc67ca48b76 P 449d4e3b09e54611ac017a3319737734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:17.974838 3528 raft_consensus.cc:729] T feb1e52541414a82b49c5cc67ca48b76 P 449d4e3b09e54611ac017a3319737734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 449d4e3b09e54611ac017a3319737734, State: Initialized, Role: FOLLOWER
I20260806 02:45:17.974740 3526 raft_consensus.cc:348] T feb1e52541414a82b49c5cc67ca48b76 P 64b51ce70efb49bf832abd4b15a5b900 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } }
I20260806 02:45:17.975039 3526 raft_consensus.cc:374] T feb1e52541414a82b49c5cc67ca48b76 P 64b51ce70efb49bf832abd4b15a5b900 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:17.975394 3528 consensus_queue.cc:260] T feb1e52541414a82b49c5cc67ca48b76 P 449d4e3b09e54611ac017a3319737734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } }
I20260806 02:45:17.975591 3526 raft_consensus.cc:729] T feb1e52541414a82b49c5cc67ca48b76 P 64b51ce70efb49bf832abd4b15a5b900 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64b51ce70efb49bf832abd4b15a5b900, State: Initialized, Role: FOLLOWER
I20260806 02:45:17.976014 3526 consensus_queue.cc:260] T feb1e52541414a82b49c5cc67ca48b76 P 64b51ce70efb49bf832abd4b15a5b900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } }
I20260806 02:45:17.976982 3555 raft_consensus.cc:2258] T b0cfd2a7839e4f549e39a11c29f46199 P e97a0e2cad6541d5a4c2a16f6ab5108d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:17.976598 3553 ts_tablet_manager.cc:1939] T b0cfd2a7839e4f549e39a11c29f46199 P 64b51ce70efb49bf832abd4b15a5b900: Deleting consensus metadata
I20260806 02:45:17.977120 3528 ts_tablet_manager.cc:1428] T feb1e52541414a82b49c5cc67ca48b76 P 449d4e3b09e54611ac017a3319737734: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260806 02:45:17.979862 3526 ts_tablet_manager.cc:1428] T feb1e52541414a82b49c5cc67ca48b76 P 64b51ce70efb49bf832abd4b15a5b900: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20260806 02:45:17.979950 2996 catalog_manager.cc:4841] TS 64b51ce70efb49bf832abd4b15a5b900 (127.1.210.67:45229): tablet b0cfd2a7839e4f549e39a11c29f46199 (table test-table [id=4932ddb9ad3c440699fd5034773c7bd1]) successfully deleted
I20260806 02:45:17.982434 3525 tablet_bootstrap.cc:492] T feb1e52541414a82b49c5cc67ca48b76 P e97a0e2cad6541d5a4c2a16f6ab5108d: No bootstrap required, opened a new log
I20260806 02:45:17.982756 3525 ts_tablet_manager.cc:1397] T feb1e52541414a82b49c5cc67ca48b76 P e97a0e2cad6541d5a4c2a16f6ab5108d: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.009s
I20260806 02:45:17.984115 3525 raft_consensus.cc:348] T feb1e52541414a82b49c5cc67ca48b76 P e97a0e2cad6541d5a4c2a16f6ab5108d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } }
I20260806 02:45:17.984431 3525 raft_consensus.cc:374] T feb1e52541414a82b49c5cc67ca48b76 P e97a0e2cad6541d5a4c2a16f6ab5108d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:17.984575 3525 raft_consensus.cc:729] T feb1e52541414a82b49c5cc67ca48b76 P e97a0e2cad6541d5a4c2a16f6ab5108d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e97a0e2cad6541d5a4c2a16f6ab5108d, State: Initialized, Role: FOLLOWER
I20260806 02:45:17.985374 3525 consensus_queue.cc:260] T feb1e52541414a82b49c5cc67ca48b76 P e97a0e2cad6541d5a4c2a16f6ab5108d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } }
I20260806 02:45:17.986254 3525 ts_tablet_manager.cc:1428] T feb1e52541414a82b49c5cc67ca48b76 P e97a0e2cad6541d5a4c2a16f6ab5108d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260806 02:45:17.988505 3555 ts_tablet_manager.cc:1905] T b0cfd2a7839e4f549e39a11c29f46199 P e97a0e2cad6541d5a4c2a16f6ab5108d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:17.991075 3554 ts_tablet_manager.cc:1918] T b0cfd2a7839e4f549e39a11c29f46199 P 449d4e3b09e54611ac017a3319737734: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:45:17.991364 3554 log.cc:1196] T b0cfd2a7839e4f549e39a11c29f46199 P 449d4e3b09e54611ac017a3319737734: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-1/wal/wals/b0cfd2a7839e4f549e39a11c29f46199
I20260806 02:45:17.992286 3554 ts_tablet_manager.cc:1939] T b0cfd2a7839e4f549e39a11c29f46199 P 449d4e3b09e54611ac017a3319737734: Deleting consensus metadata
I20260806 02:45:17.995131 2998 catalog_manager.cc:4841] TS 449d4e3b09e54611ac017a3319737734 (127.1.210.66:44591): tablet b0cfd2a7839e4f549e39a11c29f46199 (table test-table [id=4932ddb9ad3c440699fd5034773c7bd1]) successfully deleted
I20260806 02:45:17.998440 3555 ts_tablet_manager.cc:1918] T b0cfd2a7839e4f549e39a11c29f46199 P e97a0e2cad6541d5a4c2a16f6ab5108d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:45:17.998624 3555 log.cc:1196] T b0cfd2a7839e4f549e39a11c29f46199 P e97a0e2cad6541d5a4c2a16f6ab5108d: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-0/wal/wals/b0cfd2a7839e4f549e39a11c29f46199
I20260806 02:45:17.999255 3555 ts_tablet_manager.cc:1939] T b0cfd2a7839e4f549e39a11c29f46199 P e97a0e2cad6541d5a4c2a16f6ab5108d: Deleting consensus metadata
I20260806 02:45:18.001340 2998 catalog_manager.cc:4841] TS e97a0e2cad6541d5a4c2a16f6ab5108d (127.1.210.65:33949): tablet b0cfd2a7839e4f549e39a11c29f46199 (table test-table [id=4932ddb9ad3c440699fd5034773c7bd1]) successfully deleted
I20260806 02:45:18.082861 3531 raft_consensus.cc:482] T feb1e52541414a82b49c5cc67ca48b76 P 64b51ce70efb49bf832abd4b15a5b900 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:45:18.083333 3531 raft_consensus.cc:504] T feb1e52541414a82b49c5cc67ca48b76 P 64b51ce70efb49bf832abd4b15a5b900 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } }
I20260806 02:45:18.085572 3531 leader_election.cc:288] T feb1e52541414a82b49c5cc67ca48b76 P 64b51ce70efb49bf832abd4b15a5b900 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e97a0e2cad6541d5a4c2a16f6ab5108d (127.1.210.65:33949), 449d4e3b09e54611ac017a3319737734 (127.1.210.66:44591)
I20260806 02:45:18.096408 3523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:18.086863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:45073 (local address 127.1.210.66:44591)
0806 02:45:18.087083 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:18.087088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:18.087820 (+ 732us) server_negotiation.cc:407] Connection header received
0806 02:45:18.087982 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:18.087988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:18.088070 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:18.088175 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:18.089129 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.090338 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.093304 (+ 2966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.094633 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.094988 (+ 355us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:18.095122 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:18.095572 (+ 450us) server_negotiation.cc:299] Negotiation successful
0806 02:45:18.095797 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 02:45:18.096565 3520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:18.086084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:38271 (local address 127.1.210.65:33949)
0806 02:45:18.086317 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:18.086324 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:18.086621 (+ 297us) server_negotiation.cc:407] Connection header received
0806 02:45:18.087003 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:18.087009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:18.087116 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:18.087201 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:18.088128 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.089372 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.092515 (+ 3143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.094586 (+ 2071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.094896 (+ 310us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:18.095012 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:18.096005 (+ 993us) server_negotiation.cc:299] Negotiation successful
0806 02:45:18.096178 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 02:45:18.096807 3561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:18.086644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:44591 (local address 127.1.210.67:45073)
0806 02:45:18.087671 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:18.087697 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:18.087826 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:18.088215 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:18.088221 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:18.088251 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:18.088898 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:18.088923 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.090518 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.090523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:18.093071 (+ 2548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:18.093083 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.095135 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.095146 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:18.095387 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:18.095414 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 02:45:18.095543 (+ 129us) client_negotiation.cc:240] Negotiation successful
0806 02:45:18.095710 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"mutex_wait_us":1,"thread_start_us":633,"threads_started":1}
I20260806 02:45:18.097718 3560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:18.085893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:33949 (local address 127.1.210.67:38271)
0806 02:45:18.086487 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:18.086503 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:18.086794 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:18.087502 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:18.087511 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:18.087539 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:18.087954 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:18.087967 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.089510 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.089518 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:18.092236 (+ 2718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:18.092253 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.095412 (+ 3159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.095421 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:18.095708 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:18.095758 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 02:45:18.096805 (+ 1047us) client_negotiation.cc:240] Negotiation successful
0806 02:45:18.096973 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":30,"thread_start_us":292,"threads_started":1}
I20260806 02:45:18.098080 3318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feb1e52541414a82b49c5cc67ca48b76" candidate_uuid: "64b51ce70efb49bf832abd4b15a5b900" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449d4e3b09e54611ac017a3319737734" is_pre_election: true
I20260806 02:45:18.098603 3318 raft_consensus.cc:2454] T feb1e52541414a82b49c5cc67ca48b76 P 449d4e3b09e54611ac017a3319737734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64b51ce70efb49bf832abd4b15a5b900 in term 0.
I20260806 02:45:18.098583 3171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feb1e52541414a82b49c5cc67ca48b76" candidate_uuid: "64b51ce70efb49bf832abd4b15a5b900" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" is_pre_election: true
I20260806 02:45:18.098884 3171 raft_consensus.cc:2454] T feb1e52541414a82b49c5cc67ca48b76 P e97a0e2cad6541d5a4c2a16f6ab5108d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64b51ce70efb49bf832abd4b15a5b900 in term 0.
I20260806 02:45:18.099534 3388 leader_election.cc:302] T feb1e52541414a82b49c5cc67ca48b76 P 64b51ce70efb49bf832abd4b15a5b900 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 449d4e3b09e54611ac017a3319737734, 64b51ce70efb49bf832abd4b15a5b900; no voters:
I20260806 02:45:18.100244 3531 raft_consensus.cc:2781] T feb1e52541414a82b49c5cc67ca48b76 P 64b51ce70efb49bf832abd4b15a5b900 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:45:18.100499 3531 raft_consensus.cc:482] T feb1e52541414a82b49c5cc67ca48b76 P 64b51ce70efb49bf832abd4b15a5b900 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:45:18.100689 3531 raft_consensus.cc:3037] T feb1e52541414a82b49c5cc67ca48b76 P 64b51ce70efb49bf832abd4b15a5b900 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:18.103576 3531 raft_consensus.cc:504] T feb1e52541414a82b49c5cc67ca48b76 P 64b51ce70efb49bf832abd4b15a5b900 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } }
I20260806 02:45:18.104241 3531 leader_election.cc:288] T feb1e52541414a82b49c5cc67ca48b76 P 64b51ce70efb49bf832abd4b15a5b900 [CANDIDATE]: Term 1 election: Requested vote from peers e97a0e2cad6541d5a4c2a16f6ab5108d (127.1.210.65:33949), 449d4e3b09e54611ac017a3319737734 (127.1.210.66:44591)
I20260806 02:45:18.105041 3171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feb1e52541414a82b49c5cc67ca48b76" candidate_uuid: "64b51ce70efb49bf832abd4b15a5b900" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d"
I20260806 02:45:18.105296 3171 raft_consensus.cc:3037] T feb1e52541414a82b49c5cc67ca48b76 P e97a0e2cad6541d5a4c2a16f6ab5108d [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:18.105578 3318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feb1e52541414a82b49c5cc67ca48b76" candidate_uuid: "64b51ce70efb49bf832abd4b15a5b900" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449d4e3b09e54611ac017a3319737734"
I20260806 02:45:18.105942 3318 raft_consensus.cc:3037] T feb1e52541414a82b49c5cc67ca48b76 P 449d4e3b09e54611ac017a3319737734 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:18.108386 3171 raft_consensus.cc:2454] T feb1e52541414a82b49c5cc67ca48b76 P e97a0e2cad6541d5a4c2a16f6ab5108d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64b51ce70efb49bf832abd4b15a5b900 in term 1.
I20260806 02:45:18.109001 3388 leader_election.cc:302] T feb1e52541414a82b49c5cc67ca48b76 P 64b51ce70efb49bf832abd4b15a5b900 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64b51ce70efb49bf832abd4b15a5b900, e97a0e2cad6541d5a4c2a16f6ab5108d; no voters:
I20260806 02:45:18.109361 3531 raft_consensus.cc:2781] T feb1e52541414a82b49c5cc67ca48b76 P 64b51ce70efb49bf832abd4b15a5b900 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:18.109944 3531 raft_consensus.cc:686] T feb1e52541414a82b49c5cc67ca48b76 P 64b51ce70efb49bf832abd4b15a5b900 [term 1 LEADER]: Becoming Leader. State: Replica: 64b51ce70efb49bf832abd4b15a5b900, State: Running, Role: LEADER
I20260806 02:45:18.110283 3318 raft_consensus.cc:2454] T feb1e52541414a82b49c5cc67ca48b76 P 449d4e3b09e54611ac017a3319737734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64b51ce70efb49bf832abd4b15a5b900 in term 1.
I20260806 02:45:18.110800 3531 consensus_queue.cc:237] T feb1e52541414a82b49c5cc67ca48b76 P 64b51ce70efb49bf832abd4b15a5b900 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } }
I20260806 02:45:18.115337 3011 catalog_manager.cc:5495] T feb1e52541414a82b49c5cc67ca48b76 P 64b51ce70efb49bf832abd4b15a5b900 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64b51ce70efb49bf832abd4b15a5b900 (127.1.210.67). New cstate: current_term: 1 leader_uuid: "64b51ce70efb49bf832abd4b15a5b900" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } health_report { overall_health: HEALTHY } } }
Aug 06 02:45:18 dist-test-slave-8tdl krb5kdc[2966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984315, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:18.199024 3569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:18.177843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:37725 (local address 127.0.0.1:60096)
0806 02:45:18.179222 (+ 1379us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:18.179251 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:18.179452 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:18.179817 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:18.179821 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:18.181129 (+ 1308us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:18.181763 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:18.181776 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.183295 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.183299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:18.184726 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:18.184739 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.185696 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.185703 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:18.186001 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:18.186975 (+ 974us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:18.187004 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:18.190103 (+ 3099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:18.195209 (+ 5106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:18.195234 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:18.195242 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:18.195883 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:18.196399 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:18.196406 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:18.196410 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:18.196617 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:18.197329 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:18.197338 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:18.197664 (+ 326us) client_negotiation.cc:769] Sending connection context
0806 02:45:18.197911 (+ 247us) client_negotiation.cc:240] Negotiation successful
0806 02:45:18.198255 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1089,"thread_start_us":768,"threads_started":1}
I20260806 02:45:18.199461 3570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:18.177892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60096 (local address 127.1.210.126:37725)
0806 02:45:18.179530 (+ 1638us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:18.179538 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:18.179563 (+ 25us) server_negotiation.cc:407] Connection header received
0806 02:45:18.179621 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:18.179625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:18.179679 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:18.179792 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:18.182029 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.183093 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.184880 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.185450 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.185606 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:18.190418 (+ 4812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:18.190426 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:18.190431 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:18.190469 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:18.194933 (+ 4464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:18.196028 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:18.196035 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:18.196039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:18.196269 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:18.196777 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:18.196786 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:18.196789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:18.197182 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:18.197353 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:18.198007 (+ 654us) server_negotiation.cc:299] Negotiation successful
0806 02:45:18.198399 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":163,"server-negotiator.queue_time_us":1441,"thread_start_us":1268,"threads_started":1}
I20260806 02:45:18.206265 3011 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:60096:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 02:45:18.207551 3011 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:45:18.216106 3149 tablet_service.cc:1467] Processing CreateTablet for tablet 40f9da3bf59f4ec29c2b18a27e4cf5e0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=34d9b58652944f889c3720d48397140b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:45:18.216552 3291 tablet_service.cc:1467] Processing CreateTablet for tablet 40f9da3bf59f4ec29c2b18a27e4cf5e0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=34d9b58652944f889c3720d48397140b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:45:18.216784 3149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40f9da3bf59f4ec29c2b18a27e4cf5e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:18.217126 3291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40f9da3bf59f4ec29c2b18a27e4cf5e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:18.219374 3434 tablet_service.cc:1467] Processing CreateTablet for tablet 40f9da3bf59f4ec29c2b18a27e4cf5e0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=34d9b58652944f889c3720d48397140b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:45:18.220001 3434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40f9da3bf59f4ec29c2b18a27e4cf5e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:18.227525 3528 tablet_bootstrap.cc:492] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 449d4e3b09e54611ac017a3319737734: Bootstrap starting.
I20260806 02:45:18.227766 3525 tablet_bootstrap.cc:492] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P e97a0e2cad6541d5a4c2a16f6ab5108d: Bootstrap starting.
I20260806 02:45:18.229333 3528 tablet_bootstrap.cc:654] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 449d4e3b09e54611ac017a3319737734: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:18.229511 3525 tablet_bootstrap.cc:654] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P e97a0e2cad6541d5a4c2a16f6ab5108d: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:18.236018 3526 tablet_bootstrap.cc:492] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 64b51ce70efb49bf832abd4b15a5b900: Bootstrap starting.
I20260806 02:45:18.241379 3526 tablet_bootstrap.cc:654] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 64b51ce70efb49bf832abd4b15a5b900: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:18.243367 3525 tablet_bootstrap.cc:492] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P e97a0e2cad6541d5a4c2a16f6ab5108d: No bootstrap required, opened a new log
I20260806 02:45:18.243777 3525 ts_tablet_manager.cc:1397] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P e97a0e2cad6541d5a4c2a16f6ab5108d: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.010s
I20260806 02:45:18.245258 3525 raft_consensus.cc:348] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P e97a0e2cad6541d5a4c2a16f6ab5108d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } }
I20260806 02:45:18.245781 3525 raft_consensus.cc:374] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P e97a0e2cad6541d5a4c2a16f6ab5108d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:18.246011 3525 raft_consensus.cc:729] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P e97a0e2cad6541d5a4c2a16f6ab5108d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e97a0e2cad6541d5a4c2a16f6ab5108d, State: Initialized, Role: FOLLOWER
I20260806 02:45:18.246523 3525 consensus_queue.cc:260] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P e97a0e2cad6541d5a4c2a16f6ab5108d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } }
I20260806 02:45:18.247509 3533 tablet_replica.cc:439] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P e97a0e2cad6541d5a4c2a16f6ab5108d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } } }
I20260806 02:45:18.247912 3533 tablet_replica.cc:442] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P e97a0e2cad6541d5a4c2a16f6ab5108d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:45:18.248149 3525 ts_tablet_manager.cc:1428] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P e97a0e2cad6541d5a4c2a16f6ab5108d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260806 02:45:18.250233 3526 tablet_bootstrap.cc:492] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 64b51ce70efb49bf832abd4b15a5b900: No bootstrap required, opened a new log
I20260806 02:45:18.250581 3526 ts_tablet_manager.cc:1397] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 64b51ce70efb49bf832abd4b15a5b900: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.007s
I20260806 02:45:18.251240 3528 tablet_bootstrap.cc:492] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 449d4e3b09e54611ac017a3319737734: No bootstrap required, opened a new log
I20260806 02:45:18.251638 3528 ts_tablet_manager.cc:1397] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 449d4e3b09e54611ac017a3319737734: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.016s
I20260806 02:45:18.252019 3526 raft_consensus.cc:348] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 64b51ce70efb49bf832abd4b15a5b900 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } }
I20260806 02:45:18.252467 3526 raft_consensus.cc:374] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 64b51ce70efb49bf832abd4b15a5b900 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:18.252652 3526 raft_consensus.cc:729] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 64b51ce70efb49bf832abd4b15a5b900 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64b51ce70efb49bf832abd4b15a5b900, State: Initialized, Role: FOLLOWER
I20260806 02:45:18.253122 3526 consensus_queue.cc:260] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 64b51ce70efb49bf832abd4b15a5b900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } }
I20260806 02:45:18.253393 3528 raft_consensus.cc:348] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 449d4e3b09e54611ac017a3319737734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } }
I20260806 02:45:18.254014 3528 raft_consensus.cc:374] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 449d4e3b09e54611ac017a3319737734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:18.253773 3531 tablet_replica.cc:439] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 64b51ce70efb49bf832abd4b15a5b900: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } } }
I20260806 02:45:18.254196 3528 raft_consensus.cc:729] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 449d4e3b09e54611ac017a3319737734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 449d4e3b09e54611ac017a3319737734, State: Initialized, Role: FOLLOWER
I20260806 02:45:18.254305 3531 tablet_replica.cc:442] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 64b51ce70efb49bf832abd4b15a5b900: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:45:18.254616 3528 consensus_queue.cc:260] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 449d4e3b09e54611ac017a3319737734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } }
I20260806 02:45:18.255275 3539 tablet_replica.cc:439] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 449d4e3b09e54611ac017a3319737734: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } } }
I20260806 02:45:18.255697 3528 ts_tablet_manager.cc:1428] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 449d4e3b09e54611ac017a3319737734: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260806 02:45:18.255906 3539 tablet_replica.cc:442] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 449d4e3b09e54611ac017a3319737734: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:45:18.256273 3526 ts_tablet_manager.cc:1428] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 64b51ce70efb49bf832abd4b15a5b900: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260806 02:45:18.301631 3531 raft_consensus.cc:482] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 64b51ce70efb49bf832abd4b15a5b900 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:45:18.301867 3531 raft_consensus.cc:504] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 64b51ce70efb49bf832abd4b15a5b900 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } }
I20260806 02:45:18.302597 3531 leader_election.cc:288] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 64b51ce70efb49bf832abd4b15a5b900 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e97a0e2cad6541d5a4c2a16f6ab5108d (127.1.210.65:33949), 449d4e3b09e54611ac017a3319737734 (127.1.210.66:44591)
I20260806 02:45:18.303583 3171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40f9da3bf59f4ec29c2b18a27e4cf5e0" candidate_uuid: "64b51ce70efb49bf832abd4b15a5b900" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" is_pre_election: true
I20260806 02:45:18.303660 3318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40f9da3bf59f4ec29c2b18a27e4cf5e0" candidate_uuid: "64b51ce70efb49bf832abd4b15a5b900" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449d4e3b09e54611ac017a3319737734" is_pre_election: true
I20260806 02:45:18.303880 3171 raft_consensus.cc:2454] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P e97a0e2cad6541d5a4c2a16f6ab5108d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64b51ce70efb49bf832abd4b15a5b900 in term 0.
I20260806 02:45:18.303913 3318 raft_consensus.cc:2454] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 449d4e3b09e54611ac017a3319737734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64b51ce70efb49bf832abd4b15a5b900 in term 0.
I20260806 02:45:18.304473 3388 leader_election.cc:302] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 64b51ce70efb49bf832abd4b15a5b900 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 449d4e3b09e54611ac017a3319737734, 64b51ce70efb49bf832abd4b15a5b900; no voters:
I20260806 02:45:18.305043 3531 raft_consensus.cc:2781] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 64b51ce70efb49bf832abd4b15a5b900 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:45:18.305162 3531 raft_consensus.cc:482] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 64b51ce70efb49bf832abd4b15a5b900 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:45:18.305305 3531 raft_consensus.cc:3037] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 64b51ce70efb49bf832abd4b15a5b900 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:18.309736 3531 raft_consensus.cc:504] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 64b51ce70efb49bf832abd4b15a5b900 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } }
I20260806 02:45:18.310844 3531 leader_election.cc:288] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 64b51ce70efb49bf832abd4b15a5b900 [CANDIDATE]: Term 1 election: Requested vote from peers e97a0e2cad6541d5a4c2a16f6ab5108d (127.1.210.65:33949), 449d4e3b09e54611ac017a3319737734 (127.1.210.66:44591)
I20260806 02:45:18.311596 3171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40f9da3bf59f4ec29c2b18a27e4cf5e0" candidate_uuid: "64b51ce70efb49bf832abd4b15a5b900" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d"
I20260806 02:45:18.311815 3318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40f9da3bf59f4ec29c2b18a27e4cf5e0" candidate_uuid: "64b51ce70efb49bf832abd4b15a5b900" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449d4e3b09e54611ac017a3319737734"
I20260806 02:45:18.312120 3318 raft_consensus.cc:3037] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 449d4e3b09e54611ac017a3319737734 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:18.312125 3171 raft_consensus.cc:3037] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P e97a0e2cad6541d5a4c2a16f6ab5108d [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:18.316068 3318 raft_consensus.cc:2454] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 449d4e3b09e54611ac017a3319737734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64b51ce70efb49bf832abd4b15a5b900 in term 1.
I20260806 02:45:18.316699 3171 raft_consensus.cc:2454] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P e97a0e2cad6541d5a4c2a16f6ab5108d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64b51ce70efb49bf832abd4b15a5b900 in term 1.
I20260806 02:45:18.316798 3388 leader_election.cc:302] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 64b51ce70efb49bf832abd4b15a5b900 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 449d4e3b09e54611ac017a3319737734, 64b51ce70efb49bf832abd4b15a5b900; no voters:
I20260806 02:45:18.317502 3531 raft_consensus.cc:2781] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 64b51ce70efb49bf832abd4b15a5b900 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:18.317821 3531 raft_consensus.cc:686] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 64b51ce70efb49bf832abd4b15a5b900 [term 1 LEADER]: Becoming Leader. State: Replica: 64b51ce70efb49bf832abd4b15a5b900, State: Running, Role: LEADER
I20260806 02:45:18.318357 3531 consensus_queue.cc:237] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 64b51ce70efb49bf832abd4b15a5b900 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } }
I20260806 02:45:18.322497 3562 tablet_replica.cc:439] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 64b51ce70efb49bf832abd4b15a5b900: TxnStatusTablet state changed. Reason: New leader 64b51ce70efb49bf832abd4b15a5b900. Latest consensus state: current_term: 1 leader_uuid: "64b51ce70efb49bf832abd4b15a5b900" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } } }
I20260806 02:45:18.323232 3562 tablet_replica.cc:442] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 64b51ce70efb49bf832abd4b15a5b900: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:45:18.324713 3010 catalog_manager.cc:5495] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 64b51ce70efb49bf832abd4b15a5b900 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64b51ce70efb49bf832abd4b15a5b900 (127.1.210.67). New cstate: current_term: 1 leader_uuid: "64b51ce70efb49bf832abd4b15a5b900" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } health_report { overall_health: HEALTHY } } }
I20260806 02:45:18.328670 3575 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 02:45:18.329046 3575 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 02:45:18.371327 3578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:18.353313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53900 (local address 127.1.210.67:45229)
0806 02:45:18.360388 (+ 7075us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:18.360397 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:18.360463 (+ 66us) server_negotiation.cc:407] Connection header received
0806 02:45:18.360595 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:18.360601 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:18.360784 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:18.361053 (+ 269us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:18.361963 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.364132 (+ 2169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.366831 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.368008 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.368247 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:18.369177 (+ 930us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:18.370324 (+ 1147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:18.370518 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:18.370754 (+ 236us) server_negotiation.cc:299] Negotiation successful
0806 02:45:18.370851 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6717,"thread_start_us":5912,"threads_started":1}
I20260806 02:45:18.371428 3569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:18.353524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:45229 (local address 127.0.0.1:53900)
0806 02:45:18.354139 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:18.354164 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:18.354356 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:18.361193 (+ 6837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:18.361211 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:18.361239 (+ 28us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:45:18.361626 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:18.361643 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.364331 (+ 2688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.364338 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:18.366578 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:18.366596 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.368294 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.368302 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:18.368683 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:18.368778 (+ 95us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:45:18.370525 (+ 1747us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:45:18.370558 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 02:45:18.370665 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 02:45:18.370913 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371}
I20260806 02:45:18.377868 3171 raft_consensus.cc:1261] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P e97a0e2cad6541d5a4c2a16f6ab5108d [term 1 FOLLOWER]: Refusing update from remote peer 64b51ce70efb49bf832abd4b15a5b900: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:18.378059 3318 raft_consensus.cc:1261] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 449d4e3b09e54611ac017a3319737734 [term 1 FOLLOWER]: Refusing update from remote peer 64b51ce70efb49bf832abd4b15a5b900: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:18.378867 3562 consensus_queue.cc:1035] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 64b51ce70efb49bf832abd4b15a5b900 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:18.379611 3531 consensus_queue.cc:1035] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 64b51ce70efb49bf832abd4b15a5b900 [LEADER]: Connected to new peer: Peer: permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:18.394465 3533 tablet_replica.cc:439] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P e97a0e2cad6541d5a4c2a16f6ab5108d: TxnStatusTablet state changed. Reason: New leader 64b51ce70efb49bf832abd4b15a5b900. Latest consensus state: current_term: 1 leader_uuid: "64b51ce70efb49bf832abd4b15a5b900" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } } }
I20260806 02:45:18.395593 3533 tablet_replica.cc:442] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P e97a0e2cad6541d5a4c2a16f6ab5108d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:45:18.397964 3531 tablet_replica.cc:439] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 64b51ce70efb49bf832abd4b15a5b900: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "64b51ce70efb49bf832abd4b15a5b900" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } } }
I20260806 02:45:18.398440 3531 tablet_replica.cc:442] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 64b51ce70efb49bf832abd4b15a5b900: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:45:18.402601 3562 tablet_replica.cc:439] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 64b51ce70efb49bf832abd4b15a5b900: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "64b51ce70efb49bf832abd4b15a5b900" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } } }
I20260806 02:45:18.403067 3562 tablet_replica.cc:442] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 64b51ce70efb49bf832abd4b15a5b900: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:45:18.405674 3550 tablet_replica.cc:439] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 449d4e3b09e54611ac017a3319737734: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "64b51ce70efb49bf832abd4b15a5b900" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } } }
I20260806 02:45:18.406368 3550 tablet_replica.cc:442] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 449d4e3b09e54611ac017a3319737734: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:45:18.407908 3539 tablet_replica.cc:439] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 449d4e3b09e54611ac017a3319737734: TxnStatusTablet state changed. Reason: New leader 64b51ce70efb49bf832abd4b15a5b900. Latest consensus state: current_term: 1 leader_uuid: "64b51ce70efb49bf832abd4b15a5b900" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } } }
I20260806 02:45:18.408210 3539 tablet_replica.cc:442] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P 449d4e3b09e54611ac017a3319737734: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:45:18.423892 3533 tablet_replica.cc:439] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P e97a0e2cad6541d5a4c2a16f6ab5108d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "64b51ce70efb49bf832abd4b15a5b900" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 } } peers { permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 } } peers { permanent_uuid: "64b51ce70efb49bf832abd4b15a5b900" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45229 } } }
I20260806 02:45:18.424317 3533 tablet_replica.cc:442] T 40f9da3bf59f4ec29c2b18a27e4cf5e0 P e97a0e2cad6541d5a4c2a16f6ab5108d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:45:18.439440 3578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:18.431553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53908 (local address 127.1.210.67:45229)
0806 02:45:18.431770 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:18.431778 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:18.431943 (+ 165us) server_negotiation.cc:407] Connection header received
0806 02:45:18.432256 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:18.432262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:18.432367 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:18.432478 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:18.433091 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.434468 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.437089 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.437800 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.437949 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:18.438485 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:18.438626 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:18.438759 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:18.438941 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 02:45:18.439104 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 02:45:18.467099 3578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:18.457042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:47327 (local address 127.1.210.67:45229)
0806 02:45:18.457298 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:18.457305 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:18.458283 (+ 978us) server_negotiation.cc:407] Connection header received
0806 02:45:18.458606 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:18.458611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:18.458738 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:18.458832 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:18.459758 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.461367 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.464275 (+ 2908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.465300 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.465405 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:18.466233 (+ 828us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:18.466364 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:18.466436 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:18.466643 (+ 207us) server_negotiation.cc:299] Negotiation successful
0806 02:45:18.466751 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260806 02:45:18.468426 3587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:18.457068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:45229 (local address 127.1.210.67:47327)
0806 02:45:18.458163 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:18.458184 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:18.458402 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:18.459055 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:18.459060 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:18.459072 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:45:18.459559 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:18.459585 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.461595 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.461601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:18.464114 (+ 2513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:18.464131 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.465682 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.465690 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:18.465939 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:18.466030 (+ 91us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:45:18.466503 (+ 473us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:45:18.466519 (+ 16us) client_negotiation.cc:769] Sending connection context
0806 02:45:18.467594 (+ 1075us) client_negotiation.cc:240] Negotiation successful
0806 02:45:18.467755 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"thread_start_us":600,"threads_started":1}
I20260806 02:45:18.484827 3317 raft_consensus.cc:1261] T feb1e52541414a82b49c5cc67ca48b76 P 449d4e3b09e54611ac017a3319737734 [term 1 FOLLOWER]: Refusing update from remote peer 64b51ce70efb49bf832abd4b15a5b900: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:18.485838 3531 consensus_queue.cc:1035] T feb1e52541414a82b49c5cc67ca48b76 P 64b51ce70efb49bf832abd4b15a5b900 [LEADER]: Connected to new peer: Peer: permanent_uuid: "449d4e3b09e54611ac017a3319737734" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 44591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:18.486099 3171 raft_consensus.cc:1261] T feb1e52541414a82b49c5cc67ca48b76 P e97a0e2cad6541d5a4c2a16f6ab5108d [term 1 FOLLOWER]: Refusing update from remote peer 64b51ce70efb49bf832abd4b15a5b900: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:18.486840 3531 consensus_queue.cc:1035] T feb1e52541414a82b49c5cc67ca48b76 P 64b51ce70efb49bf832abd4b15a5b900 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e97a0e2cad6541d5a4c2a16f6ab5108d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:18.608886 3010 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:18.609226 3010 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:18.612622 3010 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 69e101b4d0f249a4b5cf7a908e31be9c: Sending DeleteTablet for 3 replicas of tablet feb1e52541414a82b49c5cc67ca48b76
I20260806 02:45:18.613760 3149 tablet_service.cc:1514] Processing DeleteTablet for tablet feb1e52541414a82b49c5cc67ca48b76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:18 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:42070
I20260806 02:45:18.614084 3434 tablet_service.cc:1514] Processing DeleteTablet for tablet feb1e52541414a82b49c5cc67ca48b76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:18 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:53888
I20260806 02:45:18.617111 3291 tablet_service.cc:1514] Processing DeleteTablet for tablet feb1e52541414a82b49c5cc67ca48b76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:18 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:59744
I20260806 02:45:18.627058 3594 tablet_replica.cc:330] T feb1e52541414a82b49c5cc67ca48b76 P 64b51ce70efb49bf832abd4b15a5b900: stopping tablet replica
I20260806 02:45:18.627350 3594 raft_consensus.cc:2229] T feb1e52541414a82b49c5cc67ca48b76 P 64b51ce70efb49bf832abd4b15a5b900 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:45:18.628360 3594 raft_consensus.cc:2258] T feb1e52541414a82b49c5cc67ca48b76 P 64b51ce70efb49bf832abd4b15a5b900 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:18.628489 3596 tablet_replica.cc:330] T feb1e52541414a82b49c5cc67ca48b76 P 449d4e3b09e54611ac017a3319737734: stopping tablet replica
I20260806 02:45:18.628700 3596 raft_consensus.cc:2229] T feb1e52541414a82b49c5cc67ca48b76 P 449d4e3b09e54611ac017a3319737734 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:18.628957 3596 raft_consensus.cc:2258] T feb1e52541414a82b49c5cc67ca48b76 P 449d4e3b09e54611ac017a3319737734 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:18.629685 3593 tablet_replica.cc:330] T feb1e52541414a82b49c5cc67ca48b76 P e97a0e2cad6541d5a4c2a16f6ab5108d: stopping tablet replica
I20260806 02:45:18.629980 3593 raft_consensus.cc:2229] T feb1e52541414a82b49c5cc67ca48b76 P e97a0e2cad6541d5a4c2a16f6ab5108d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:18.630357 3593 raft_consensus.cc:2258] T feb1e52541414a82b49c5cc67ca48b76 P e97a0e2cad6541d5a4c2a16f6ab5108d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:18.630373 3596 ts_tablet_manager.cc:1905] T feb1e52541414a82b49c5cc67ca48b76 P 449d4e3b09e54611ac017a3319737734: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:18.632321 3594 ts_tablet_manager.cc:1905] T feb1e52541414a82b49c5cc67ca48b76 P 64b51ce70efb49bf832abd4b15a5b900: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:18.632658 3593 ts_tablet_manager.cc:1905] T feb1e52541414a82b49c5cc67ca48b76 P e97a0e2cad6541d5a4c2a16f6ab5108d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:18.643870 3594 ts_tablet_manager.cc:1918] T feb1e52541414a82b49c5cc67ca48b76 P 64b51ce70efb49bf832abd4b15a5b900: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:45:18.644170 3594 log.cc:1196] T feb1e52541414a82b49c5cc67ca48b76 P 64b51ce70efb49bf832abd4b15a5b900: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-2/wal/wals/feb1e52541414a82b49c5cc67ca48b76
I20260806 02:45:18.644822 3594 ts_tablet_manager.cc:1939] T feb1e52541414a82b49c5cc67ca48b76 P 64b51ce70efb49bf832abd4b15a5b900: Deleting consensus metadata
I20260806 02:45:18.646256 2996 catalog_manager.cc:4841] TS 64b51ce70efb49bf832abd4b15a5b900 (127.1.210.67:45229): tablet feb1e52541414a82b49c5cc67ca48b76 (table test-table [id=87d9ec117dd74ac3b657470ed5ebd880]) successfully deleted
I20260806 02:45:18.648607 3596 ts_tablet_manager.cc:1918] T feb1e52541414a82b49c5cc67ca48b76 P 449d4e3b09e54611ac017a3319737734: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:45:18.648813 3596 log.cc:1196] T feb1e52541414a82b49c5cc67ca48b76 P 449d4e3b09e54611ac017a3319737734: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-1/wal/wals/feb1e52541414a82b49c5cc67ca48b76
I20260806 02:45:18.649458 3596 ts_tablet_manager.cc:1939] T feb1e52541414a82b49c5cc67ca48b76 P 449d4e3b09e54611ac017a3319737734: Deleting consensus metadata
I20260806 02:45:18.651139 2998 catalog_manager.cc:4841] TS 449d4e3b09e54611ac017a3319737734 (127.1.210.66:44591): tablet feb1e52541414a82b49c5cc67ca48b76 (table test-table [id=87d9ec117dd74ac3b657470ed5ebd880]) successfully deleted
I20260806 02:45:18.663970 3593 ts_tablet_manager.cc:1918] T feb1e52541414a82b49c5cc67ca48b76 P e97a0e2cad6541d5a4c2a16f6ab5108d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:45:18.664146 3593 log.cc:1196] T feb1e52541414a82b49c5cc67ca48b76 P e97a0e2cad6541d5a4c2a16f6ab5108d: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984304136196-1865-0/minicluster-data/ts-0/wal/wals/feb1e52541414a82b49c5cc67ca48b76
I20260806 02:45:18.664690 3593 ts_tablet_manager.cc:1939] T feb1e52541414a82b49c5cc67ca48b76 P e97a0e2cad6541d5a4c2a16f6ab5108d: Deleting consensus metadata
I20260806 02:45:18.666092 2998 catalog_manager.cc:4841] TS e97a0e2cad6541d5a4c2a16f6ab5108d (127.1.210.65:33949): tablet feb1e52541414a82b49c5cc67ca48b76 (table test-table [id=87d9ec117dd74ac3b657470ed5ebd880]) successfully deleted
Aug 06 02:45:18 dist-test-slave-8tdl krb5kdc[2966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984317, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.210.65@KRBTEST.COM
I20260806 02:45:18.685550 3597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:18.634061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42072 (local address 127.1.210.65:33949)
0806 02:45:18.657125 (+ 23064us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:18.657133 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:18.657156 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:45:18.657239 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:18.657244 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:18.657384 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:18.657552 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:18.661091 (+ 3539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.662846 (+ 1755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.665976 (+ 3130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.667215 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.667382 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:18.675561 (+ 8179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:18.675583 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:18.675605 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:18.675703 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:18.680717 (+ 5014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:18.682188 (+ 1471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:18.682199 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:18.682203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:18.682648 (+ 445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:18.683380 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:18.683387 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:18.683391 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:18.683975 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:18.684367 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:18.684721 (+ 354us) server_negotiation.cc:299] Negotiation successful
0806 02:45:18.684990 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22874,"thread_start_us":23619,"threads_started":1}
W20260806 02:45:18.687021 3216 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42072
I20260806 02:45:18.712244 3570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:18.691285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60100 (local address 127.1.210.126:37725)
0806 02:45:18.691613 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:18.691619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:18.692383 (+ 764us) server_negotiation.cc:407] Connection header received
0806 02:45:18.692545 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:18.692552 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:18.692652 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:18.692770 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:18.694908 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.696197 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.698363 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.699162 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.699353 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:18.703209 (+ 3856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:18.703228 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:18.703233 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:18.703283 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:18.708460 (+ 5177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:18.709561 (+ 1101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:18.709568 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:18.709571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:18.709726 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:18.710144 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:18.710149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:18.710154 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:18.710723 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:18.711078 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:18.711334 (+ 256us) server_negotiation.cc:299] Negotiation successful
0806 02:45:18.711643 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
W20260806 02:45:18.713487 3010 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60100
I20260806 02:45:18.716183 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 3084
I20260806 02:45:18.738300 3597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:18.717434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42076 (local address 127.1.210.65:33949)
0806 02:45:18.717659 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:18.717666 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:18.720078 (+ 2412us) server_negotiation.cc:407] Connection header received
0806 02:45:18.720301 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:18.720307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:18.720601 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:18.720727 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:18.722225 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.723453 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.725257 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.725836 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.725963 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:18.729324 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:18.729332 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:18.729336 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:18.729381 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:18.734483 (+ 5102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:18.735524 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:18.735529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:18.735533 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:18.735776 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:18.736395 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:18.736400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:18.736404 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:18.736809 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:18.737103 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:18.737595 (+ 492us) server_negotiation.cc:299] Negotiation successful
0806 02:45:18.737889 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260806 02:45:18.739149 3216 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42076
W20260806 02:45:18.739904 1865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 3084: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:45:18.740245 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 3084
I20260806 02:45:18.759356 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 3225
Aug 06 02:45:18 dist-test-slave-8tdl krb5kdc[2966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984317, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.210.66@KRBTEST.COM
I20260806 02:45:18.778558 3602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:18.760131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59758 (local address 127.1.210.66:44591)
0806 02:45:18.760681 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:18.760689 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:18.760711 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:45:18.760782 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:18.760788 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:18.760867 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:18.760997 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:18.762606 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.763375 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.765368 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.765886 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.766009 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:18.771573 (+ 5564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:18.771591 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:18.771607 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:18.771657 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:18.775584 (+ 3927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:18.776391 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:18.776399 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:18.776402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:18.776549 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:18.776948 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:18.776956 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:18.776960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:18.777314 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:18.777637 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:18.777934 (+ 297us) server_negotiation.cc:299] Negotiation successful
0806 02:45:18.778176 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":274,"threads_started":1}
W20260806 02:45:18.779428 3359 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59758
W20260806 02:45:18.780145 1865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 3225: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:45:18.780354 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 3225
I20260806 02:45:18.802611 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 3368
Aug 06 02:45:18 dist-test-slave-8tdl krb5kdc[2966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984317, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.210.67@KRBTEST.COM
I20260806 02:45:18.821288 3578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:18.803534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53922 (local address 127.1.210.67:45229)
0806 02:45:18.803737 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:18.803744 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:18.803767 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:45:18.803971 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:18.803977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:18.804075 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:18.804204 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:18.805492 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.806608 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.808162 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.808912 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.809071 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:18.814197 (+ 5126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:18.814218 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:18.814240 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:18.814294 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:18.817995 (+ 3701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:18.818997 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:18.819013 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:18.819015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:18.819219 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:18.819662 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:18.819665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:18.819667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:18.820107 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:18.820372 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:18.820770 (+ 398us) server_negotiation.cc:299] Negotiation successful
0806 02:45:18.820990 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260806 02:45:18.822065 3500 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53922
W20260806 02:45:18.822875 1865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 3368: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:45:18.823063 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 3368
I20260806 02:45:18.843900 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 2979
I20260806 02:45:18.845244 3053 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:45:18.963110 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 2979
2026-08-06T02:45:18Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5263 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:45:19 dist-test-slave-8tdl krb5kdc[3606](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:45:19 dist-test-slave-8tdl krb5kdc[3606](info): set up 2 sockets
Aug 06 02:45:19 dist-test-slave-8tdl krb5kdc[3606](info): commencing operation
krb5kdc: starting...
W20260806 02:45:20.102342 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:45:20 dist-test-slave-8tdl krb5kdc[3606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:45:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
I20260806 02:45:20.362599 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:40885
--webserver_interface=127.1.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:36339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:40885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:20.709367 3619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:20.709692 3619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:20.709864 3619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:20.717443 3619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:20.717566 3619 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 02:45:20.717659 3619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:20.717777 3619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:20.717866 3619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:20.717985 3619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:45:20.728811 3619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:36339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:40885
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:40885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:20.730334 3619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:20.731822 3619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:20.744959 3625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:20.744872 3624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:20.744915 3627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:20.746003 3619 server_base.cc:1034] running on GCE node
I20260806 02:45:20.747025 3619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:20.748675 3619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:20.749953 3619 hybrid_clock.cc:648] HybridClock initialized: now 1785984320749883 us; error 39 us; skew 500 ppm
I20260806 02:45:20.750429 3619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:45:20 dist-test-slave-8tdl krb5kdc[3606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984320, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:20.753954 3619 init.cc:377] Logged in from keytab as kudu/127.1.210.126@KRBTEST.COM (short username kudu)
I20260806 02:45:20.755790 3619 webserver.cc:460] Webserver started at http://127.1.210.126:44039/ using document root <none> and password file <none>
I20260806 02:45:20.756568 3619 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:20.756661 3619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:20.757023 3619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:20.760324 3619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "4934c13314b0457a895cea460da2b23d"
format_stamp: "Formatted at 2026-08-06 02:45:20 on dist-test-slave-8tdl"
server_key: "d6c0a26cdd21c635343e427ab641e3b0"
server_key_iv: "d59e24715fa6d65e5df6d9257172f8b9"
server_key_version: "encryptionkey@0"
I20260806 02:45:20.761195 3619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "4934c13314b0457a895cea460da2b23d"
format_stamp: "Formatted at 2026-08-06 02:45:20 on dist-test-slave-8tdl"
server_key: "d6c0a26cdd21c635343e427ab641e3b0"
server_key_iv: "d59e24715fa6d65e5df6d9257172f8b9"
server_key_version: "encryptionkey@0"
I20260806 02:45:20.767216 3619 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 02:45:20.771359 3634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:20.773103 3619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:45:20.773409 3619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "4934c13314b0457a895cea460da2b23d"
format_stamp: "Formatted at 2026-08-06 02:45:20 on dist-test-slave-8tdl"
server_key: "d6c0a26cdd21c635343e427ab641e3b0"
server_key_iv: "d59e24715fa6d65e5df6d9257172f8b9"
server_key_version: "encryptionkey@0"
I20260806 02:45:20.773742 3619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:20.801003 3619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:20.805900 3619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:20.806280 3619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:20.829900 3619 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:40885
I20260806 02:45:20.829881 3697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:40885 every 8 connection(s)
I20260806 02:45:20.831419 3619 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:45:20.834748 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 3619
I20260806 02:45:20.835000 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:45:20.835821 1865 external_mini_cluster.cc:1428] Setting key fcea8846f70bec1f1e1468509c6bc99a
I20260806 02:45:20.836820 3698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:20.850474 3698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4934c13314b0457a895cea460da2b23d: Bootstrap starting.
Aug 06 02:45:20 dist-test-slave-8tdl krb5kdc[3606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:20.853915 3698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4934c13314b0457a895cea460da2b23d: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:20.855199 3698 log.cc:824] T 00000000000000000000000000000000 P 4934c13314b0457a895cea460da2b23d: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:20.858847 3698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4934c13314b0457a895cea460da2b23d: No bootstrap required, opened a new log
I20260806 02:45:20.860699 3701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:20.838208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36420 (local address 127.1.210.126:40885)
0806 02:45:20.839187 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:20.839200 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:20.839244 (+ 44us) server_negotiation.cc:407] Connection header received
0806 02:45:20.841815 (+ 2571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:20.841842 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:20.842306 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:20.842827 (+ 521us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:20.844430 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:20.845779 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:20.847229 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:20.848044 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:20.848299 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:20.852594 (+ 4295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:20.852603 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:20.852619 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:20.852660 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:20.856123 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:20.856959 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:20.856967 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:20.856974 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:20.857254 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:20.857713 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:20.857717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:20.857719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:20.858280 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:20.858630 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:20.859160 (+ 530us) server_negotiation.cc:299] Negotiation successful
0806 02:45:20.859637 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725,"thread_start_us":505,"threads_started":1}
I20260806 02:45:20.867286 3698 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4934c13314b0457a895cea460da2b23d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4934c13314b0457a895cea460da2b23d" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 40885 } }
I20260806 02:45:20.867825 3698 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4934c13314b0457a895cea460da2b23d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:20.867964 3698 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4934c13314b0457a895cea460da2b23d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4934c13314b0457a895cea460da2b23d, State: Initialized, Role: FOLLOWER
I20260806 02:45:20.868633 3698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4934c13314b0457a895cea460da2b23d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4934c13314b0457a895cea460da2b23d" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 40885 } }
I20260806 02:45:20.868848 3698 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4934c13314b0457a895cea460da2b23d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:45:20.868949 3698 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4934c13314b0457a895cea460da2b23d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:45:20.869074 3698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4934c13314b0457a895cea460da2b23d [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:20.871330 3698 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4934c13314b0457a895cea460da2b23d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4934c13314b0457a895cea460da2b23d" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 40885 } }
I20260806 02:45:20.871800 3698 leader_election.cc:302] T 00000000000000000000000000000000 P 4934c13314b0457a895cea460da2b23d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4934c13314b0457a895cea460da2b23d; no voters:
I20260806 02:45:20.872395 3698 leader_election.cc:288] T 00000000000000000000000000000000 P 4934c13314b0457a895cea460da2b23d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:45:20.872610 3703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4934c13314b0457a895cea460da2b23d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:20.873392 3703 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4934c13314b0457a895cea460da2b23d [term 1 LEADER]: Becoming Leader. State: Replica: 4934c13314b0457a895cea460da2b23d, State: Running, Role: LEADER
I20260806 02:45:20.874053 3703 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4934c13314b0457a895cea460da2b23d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4934c13314b0457a895cea460da2b23d" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 40885 } }
I20260806 02:45:20.874828 3698 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4934c13314b0457a895cea460da2b23d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:20.877588 3705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4934c13314b0457a895cea460da2b23d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4934c13314b0457a895cea460da2b23d. Latest consensus state: current_term: 1 leader_uuid: "4934c13314b0457a895cea460da2b23d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4934c13314b0457a895cea460da2b23d" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 40885 } } }
I20260806 02:45:20.877920 3704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4934c13314b0457a895cea460da2b23d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4934c13314b0457a895cea460da2b23d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4934c13314b0457a895cea460da2b23d" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 40885 } } }
I20260806 02:45:20.878134 3705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4934c13314b0457a895cea460da2b23d [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:20.878247 3704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4934c13314b0457a895cea460da2b23d [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:20.880031 3711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:20.885872 3711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:45:20.896006 3711 catalog_manager.cc:1347] Generated new cluster ID: a85bb4f03b504ac6914fdfbe06099f36
I20260806 02:45:20.896148 3711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:20.923256 3711 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:45:20.924813 3711 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:20.937498 3711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4934c13314b0457a895cea460da2b23d: Generated new TSK 0
I20260806 02:45:20.938725 3711 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab.
Entry for principal kudu/127.1.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab.
I20260806 02:45:20.995221 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.65:0
--local_ip_for_outbound_sockets=127.1.210.65
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:40885
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:36339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:21.342855 3724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:21.343152 3724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:21.343330 3724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:21.350970 3724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:21.351094 3724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:21.351253 3724 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 02:45:21.351315 3724 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:45:21.351424 3724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.65
I20260806 02:45:21.362443 3724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:36339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.210.126:40885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.65
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:21.363873 3724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:21.365275 3724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:21.378844 3730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:21.378970 3729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:21.378932 3732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:21.380263 3724 server_base.cc:1034] running on GCE node
I20260806 02:45:21.381011 3724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:21.381984 3724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:21.383219 3724 hybrid_clock.cc:648] HybridClock initialized: now 1785984321383163 us; error 48 us; skew 500 ppm
I20260806 02:45:21.383549 3724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:45:21 dist-test-slave-8tdl krb5kdc[3606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984321, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:21.392263 3724 init.cc:377] Logged in from keytab as kudu/127.1.210.65@KRBTEST.COM (short username kudu)
I20260806 02:45:21.394356 3724 webserver.cc:460] Webserver started at http://127.1.210.65:37377/ using document root <none> and password file <none>
I20260806 02:45:21.395203 3724 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:21.395417 3724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:21.395761 3724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:21.398852 3724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-0/data/instance:
uuid: "4c122a1f079c4ce78297526b89f5f60a"
format_stamp: "Formatted at 2026-08-06 02:45:21 on dist-test-slave-8tdl"
server_key: "2c48a206bd0e4ca41e21276373108132"
server_key_iv: "ed46c331f830ee83c67fc0b3c4a8de70"
server_key_version: "encryptionkey@0"
I20260806 02:45:21.399678 3724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance:
uuid: "4c122a1f079c4ce78297526b89f5f60a"
format_stamp: "Formatted at 2026-08-06 02:45:21 on dist-test-slave-8tdl"
server_key: "2c48a206bd0e4ca41e21276373108132"
server_key_iv: "ed46c331f830ee83c67fc0b3c4a8de70"
server_key_version: "encryptionkey@0"
I20260806 02:45:21.405119 3724 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 02:45:21.408990 3739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:21.410476 3724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:45:21.410661 3724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-0/wal
uuid: "4c122a1f079c4ce78297526b89f5f60a"
format_stamp: "Formatted at 2026-08-06 02:45:21 on dist-test-slave-8tdl"
server_key: "2c48a206bd0e4ca41e21276373108132"
server_key_iv: "ed46c331f830ee83c67fc0b3c4a8de70"
server_key_version: "encryptionkey@0"
I20260806 02:45:21.410949 3724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:21.430112 3724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:21.434801 3724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:21.435115 3724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:21.438128 3724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:21.438261 3724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:21.438400 3724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:21.438495 3724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:45:21 dist-test-slave-8tdl krb5kdc[3606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984321, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:21.470449 3749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:21.439852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:40885 (local address 127.1.210.65:48339)
0806 02:45:21.441087 (+ 1235us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:21.441151 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:21.444416 (+ 3265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:21.445469 (+ 1053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:21.445482 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:21.446605 (+ 1123us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:21.447842 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:21.447882 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.449354 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.449365 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:21.452404 (+ 3039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:21.452429 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.453323 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.453331 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:21.453759 (+ 428us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:21.454759 (+ 1000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:21.454801 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:21.458720 (+ 3919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:21.463566 (+ 4846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:21.463582 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:21.463616 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:21.464522 (+ 906us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:21.465062 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:21.465081 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:21.465083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:21.465486 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:21.466154 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:21.466166 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:21.466966 (+ 800us) client_negotiation.cc:769] Sending connection context
0806 02:45:21.467405 (+ 439us) client_negotiation.cc:240] Negotiation successful
0806 02:45:21.468221 (+ 816us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":878,"thread_start_us":597,"threads_started":1}
I20260806 02:45:21.471031 3748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:21.439014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:48339 (local address 127.1.210.126:40885)
0806 02:45:21.442101 (+ 3087us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:21.442106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:21.442127 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:45:21.444909 (+ 2782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:21.444915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:21.445000 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:21.445098 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:21.448122 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.449138 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.452610 (+ 3472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.453057 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.453209 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:21.459082 (+ 5873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:21.459099 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:21.459102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:21.459140 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:21.463294 (+ 4154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:21.464737 (+ 1443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:21.464744 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:21.464748 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:21.464923 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:21.465669 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:21.465673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:21.465676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:21.465968 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:21.466094 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:21.470321 (+ 4227us) server_negotiation.cc:299] Negotiation successful
0806 02:45:21.470617 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2965,"thread_start_us":2869,"threads_started":1}
I20260806 02:45:21.486959 3724 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.65:46461
I20260806 02:45:21.487075 3861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.65:46461 every 8 connection(s)
I20260806 02:45:21.488768 3724 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
I20260806 02:45:21.491380 3862 heartbeater.cc:347] Connected to a master server at 127.1.210.126:40885
I20260806 02:45:21.491657 3862 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:21.492341 3862 heartbeater.cc:511] Master 127.1.210.126:40885 requested a full tablet report, sending...
I20260806 02:45:21.495296 3653 ts_manager.cc:194] Registered new tserver with Master: 4c122a1f079c4ce78297526b89f5f60a (127.1.210.65:46461)
I20260806 02:45:21.495632 3841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:21.478184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:40885 (local address 127.1.210.65:47181)
0806 02:45:21.478899 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:21.478919 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:21.479096 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:21.479857 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:21.479864 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:21.480892 (+ 1028us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:21.481246 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:21.481256 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.483014 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.483025 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:21.485165 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:21.485185 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.486511 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.486522 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:21.486795 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:21.487555 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:21.487576 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:21.488480 (+ 904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:21.492588 (+ 4108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:21.492595 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:21.492602 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:21.493223 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:21.493728 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:21.493732 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:21.493735 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:21.493893 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:21.494612 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:21.494615 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:21.494839 (+ 224us) client_negotiation.cc:769] Sending connection context
0806 02:45:21.494976 (+ 137us) client_negotiation.cc:240] Negotiation successful
0806 02:45:21.495257 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":691,"threads_started":1}
I20260806 02:45:21.495805 3748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:21.478622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:47181 (local address 127.1.210.126:40885)
0806 02:45:21.478863 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:21.478869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:21.479041 (+ 172us) server_negotiation.cc:407] Connection header received
0806 02:45:21.479267 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:21.479277 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:21.479419 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:21.479588 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:21.481445 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.482825 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.485392 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.486254 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.486439 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:21.488722 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:21.488729 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:21.488733 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:21.488771 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:21.492421 (+ 3650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:21.493363 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:21.493369 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:21.493372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:21.493607 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:21.494020 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:21.494028 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:21.494032 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:21.494474 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:21.494644 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:21.495138 (+ 494us) server_negotiation.cc:299] Negotiation successful
0806 02:45:21.495420 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260806 02:45:21.497181 3653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.65@KRBTEST.COM'} at 127.1.210.65:48339
I20260806 02:45:21.497670 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 3724
I20260806 02:45:21.497891 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance
I20260806 02:45:21.498833 1865 external_mini_cluster.cc:1428] Setting key 0662882c9724668e340b0d49593aab18
WARNING: no policy specified for kudu/127.1.210.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab.
Entry for principal kudu/127.1.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab.
I20260806 02:45:21.550583 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.66:0
--local_ip_for_outbound_sockets=127.1.210.66
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:40885
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:36339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:22.165992 3866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:22.166792 3866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:22.167146 3866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:22.184015 3866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:22.184356 3866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:22.184670 3866 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 02:45:22.184803 3866 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:45:22.185043 3866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.66
I20260806 02:45:22.208981 3866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:36339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.210.126:40885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.66
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:22.211500 3866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:22.215263 3866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:22.246317 3871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:22.247264 3874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:22.247308 3872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:22.248164 3866 server_base.cc:1034] running on GCE node
I20260806 02:45:22.249588 3866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:22.251228 3866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:22.252846 3866 hybrid_clock.cc:648] HybridClock initialized: now 1785984322252720 us; error 63 us; skew 500 ppm
I20260806 02:45:22.253681 3866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:45:22 dist-test-slave-8tdl krb5kdc[3606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984322, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:22.260190 3866 init.cc:377] Logged in from keytab as kudu/127.1.210.66@KRBTEST.COM (short username kudu)
I20260806 02:45:22.263335 3866 webserver.cc:460] Webserver started at http://127.1.210.66:40875/ using document root <none> and password file <none>
I20260806 02:45:22.264717 3866 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:22.264928 3866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:22.265566 3866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:22.272374 3866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-1/data/instance:
uuid: "87d16739d49043fdb3241c9486c4292e"
format_stamp: "Formatted at 2026-08-06 02:45:22 on dist-test-slave-8tdl"
server_key: "4321a797c51e8b123f3d260acfce7464"
server_key_iv: "feb6f0fb7f4b9b88cbc1f386572a8c47"
server_key_version: "encryptionkey@0"
I20260806 02:45:22.275211 3866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance:
uuid: "87d16739d49043fdb3241c9486c4292e"
format_stamp: "Formatted at 2026-08-06 02:45:22 on dist-test-slave-8tdl"
server_key: "4321a797c51e8b123f3d260acfce7464"
server_key_iv: "feb6f0fb7f4b9b88cbc1f386572a8c47"
server_key_version: "encryptionkey@0"
I20260806 02:45:22.288064 3866 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.006s sys 0.005s
I20260806 02:45:22.297873 3881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:22.301735 3866 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.004s sys 0.002s
I20260806 02:45:22.302042 3866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-1/wal
uuid: "87d16739d49043fdb3241c9486c4292e"
format_stamp: "Formatted at 2026-08-06 02:45:22 on dist-test-slave-8tdl"
server_key: "4321a797c51e8b123f3d260acfce7464"
server_key_iv: "feb6f0fb7f4b9b88cbc1f386572a8c47"
server_key_version: "encryptionkey@0"
I20260806 02:45:22.302488 3866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:22.346406 3866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:22.358668 3866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:22.359202 3866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:22.365597 3866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:22.365859 3866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:22.366148 3866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:22.366530 3866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.002s sys 0.000s
Aug 06 02:45:22 dist-test-slave-8tdl krb5kdc[3606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984322, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:22.428313 3889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:22.366746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:58299 (local address 127.1.210.126:40885)
0806 02:45:22.367906 (+ 1160us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:22.367927 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:22.372232 (+ 4305us) server_negotiation.cc:407] Connection header received
0806 02:45:22.378037 (+ 5805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:22.378044 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:22.378569 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:22.379079 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:22.385299 (+ 6220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.388570 (+ 3271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.394225 (+ 5655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.396152 (+ 1927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.396483 (+ 331us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:22.406028 (+ 9545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:22.406040 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:22.406049 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:22.406157 (+ 108us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:22.416168 (+ 10011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:22.419021 (+ 2853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:22.419042 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:22.419058 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:22.419428 (+ 370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:22.421088 (+ 1660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:22.421105 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:22.421108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:22.422183 (+ 1075us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:22.422570 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:22.425321 (+ 2751us) server_negotiation.cc:299] Negotiation successful
0806 02:45:22.426394 (+ 1073us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":789,"thread_start_us":494,"threads_started":1}
I20260806 02:45:22.429028 3891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:22.369048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:40885 (local address 127.1.210.66:58299)
0806 02:45:22.371787 (+ 2739us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:22.371910 (+ 123us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:22.377248 (+ 5338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:22.379802 (+ 2554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:22.379873 (+ 71us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:22.382520 (+ 2647us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:22.384487 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:22.384576 (+ 89us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.389242 (+ 4666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.389263 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:22.393793 (+ 4530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:22.393833 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.396852 (+ 3019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.396884 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:22.397288 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:22.398930 (+ 1642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:22.399150 (+ 220us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:22.405306 (+ 6156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:22.416692 (+ 11386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:22.416704 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:22.416803 (+ 99us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:22.418306 (+ 1503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:22.419911 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:22.419942 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:22.419946 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:22.420827 (+ 881us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:22.422519 (+ 1692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:22.422536 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:22.424011 (+ 1475us) client_negotiation.cc:769] Sending connection context
0806 02:45:22.425005 (+ 994us) client_negotiation.cc:240] Negotiation successful
0806 02:45:22.426014 (+ 1009us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1744,"thread_start_us":1228,"threads_started":1}
I20260806 02:45:22.490039 3889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:22.438136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:57189 (local address 127.1.210.126:40885)
0806 02:45:22.438604 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:22.438624 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:22.441446 (+ 2822us) server_negotiation.cc:407] Connection header received
0806 02:45:22.442184 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:22.442204 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:22.442467 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:22.443184 (+ 717us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:22.445506 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.450482 (+ 4976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.457009 (+ 6527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.459200 (+ 2191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.459938 (+ 738us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:22.468526 (+ 8588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:22.468546 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:22.468560 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:22.468731 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:22.478050 (+ 9319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:22.481165 (+ 3115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:22.481174 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:22.481178 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:22.481481 (+ 303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:22.483265 (+ 1784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:22.483277 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:22.483280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:22.484394 (+ 1114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:22.484699 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:22.487319 (+ 2620us) server_negotiation.cc:299] Negotiation successful
0806 02:45:22.487901 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260806 02:45:22.492534 3966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:22.439419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:40885 (local address 127.1.210.66:57189)
0806 02:45:22.441089 (+ 1670us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:22.441157 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:22.441506 (+ 349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:22.443102 (+ 1596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:22.443125 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:22.444172 (+ 1047us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:22.444874 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:22.444963 (+ 89us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.451197 (+ 6234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.451217 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:22.456537 (+ 5320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:22.456567 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.460185 (+ 3618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.460342 (+ 157us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:22.461848 (+ 1506us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:22.464278 (+ 2430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:22.464320 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:22.467722 (+ 3402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:22.478454 (+ 10732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:22.478465 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:22.478473 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:22.480736 (+ 2263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:22.481809 (+ 1073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:22.481851 (+ 42us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:22.481891 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:22.482962 (+ 1071us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:22.484753 (+ 1791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:22.484773 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:22.486120 (+ 1347us) client_negotiation.cc:769] Sending connection context
0806 02:45:22.489783 (+ 3663us) client_negotiation.cc:240] Negotiation successful
0806 02:45:22.491057 (+ 1274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1217,"thread_start_us":926,"threads_started":1}
I20260806 02:45:22.495126 3866 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.66:35359
I20260806 02:45:22.495289 4004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.66:35359 every 8 connection(s)
I20260806 02:45:22.497974 3866 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
I20260806 02:45:22.502344 3862 heartbeater.cc:503] Master 127.1.210.126:40885 was elected leader, sending a full tablet report...
I20260806 02:45:22.506278 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 3866
I20260806 02:45:22.507004 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance
I20260806 02:45:22.507504 4005 heartbeater.cc:347] Connected to a master server at 127.1.210.126:40885
I20260806 02:45:22.508219 4005 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:22.509262 1865 external_mini_cluster.cc:1428] Setting key 690b8dbdef34a13815170c20e5e45e4e
I20260806 02:45:22.511416 4005 heartbeater.cc:511] Master 127.1.210.126:40885 requested a full tablet report, sending...
I20260806 02:45:22.514626 3651 ts_manager.cc:194] Registered new tserver with Master: 87d16739d49043fdb3241c9486c4292e (127.1.210.66:35359)
I20260806 02:45:22.518150 3651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.66@KRBTEST.COM'} at 127.1.210.66:58299
WARNING: no policy specified for kudu/127.1.210.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab.
Entry for principal kudu/127.1.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab.
I20260806 02:45:22.596724 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.67:0
--local_ip_for_outbound_sockets=127.1.210.67
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:40885
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:36339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:23.134351 4009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:23.134972 4009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:23.135229 4009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:23.147281 4009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:23.147450 4009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:23.147729 4009 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 02:45:23.147816 4009 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:45:23.147982 4009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.67
I20260806 02:45:23.162955 4009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:36339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.210.126:40885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.67
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:23.164942 4009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:23.166764 4009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:23.189595 4015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:23.189509 4014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:23.189687 4017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:23.192202 4009 server_base.cc:1034] running on GCE node
I20260806 02:45:23.193344 4009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:23.195313 4009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:23.196781 4009 hybrid_clock.cc:648] HybridClock initialized: now 1785984323196685 us; error 118 us; skew 500 ppm
I20260806 02:45:23.197326 4009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:45:23 dist-test-slave-8tdl krb5kdc[3606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984323, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:23.202735 4009 init.cc:377] Logged in from keytab as kudu/127.1.210.67@KRBTEST.COM (short username kudu)
I20260806 02:45:23.205687 4009 webserver.cc:460] Webserver started at http://127.1.210.67:46277/ using document root <none> and password file <none>
I20260806 02:45:23.208225 4009 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:23.208740 4009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:23.209475 4009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:23.216312 4009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-2/data/instance:
uuid: "8af34451093b451097f4352ab317f2ae"
format_stamp: "Formatted at 2026-08-06 02:45:23 on dist-test-slave-8tdl"
server_key: "0bdc8ecd993aaa6de7e4451e9c429b19"
server_key_iv: "7a5ef99a9a07b10c88bbb3a0fae030fd"
server_key_version: "encryptionkey@0"
I20260806 02:45:23.218190 4009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance:
uuid: "8af34451093b451097f4352ab317f2ae"
format_stamp: "Formatted at 2026-08-06 02:45:23 on dist-test-slave-8tdl"
server_key: "0bdc8ecd993aaa6de7e4451e9c429b19"
server_key_iv: "7a5ef99a9a07b10c88bbb3a0fae030fd"
server_key_version: "encryptionkey@0"
I20260806 02:45:23.228411 4009 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.006s sys 0.001s
I20260806 02:45:23.234205 4024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:23.236989 4009 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260806 02:45:23.237327 4009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-2/wal
uuid: "8af34451093b451097f4352ab317f2ae"
format_stamp: "Formatted at 2026-08-06 02:45:23 on dist-test-slave-8tdl"
server_key: "0bdc8ecd993aaa6de7e4451e9c429b19"
server_key_iv: "7a5ef99a9a07b10c88bbb3a0fae030fd"
server_key_version: "encryptionkey@0"
I20260806 02:45:23.237645 4009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:23.287868 4009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:23.296324 4009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:23.297504 4009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:23.303689 4009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:23.303920 4009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:23.304085 4009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:23.304184 4009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:45:23 dist-test-slave-8tdl krb5kdc[3606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984323, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:23.352569 4033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:23.304584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:45445 (local address 127.1.210.126:40885)
0806 02:45:23.305293 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:23.305298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:23.307293 (+ 1995us) server_negotiation.cc:407] Connection header received
0806 02:45:23.311542 (+ 4249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:23.311549 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:23.311705 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:23.312034 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:23.316383 (+ 4349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:23.318838 (+ 2455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:23.324551 (+ 5713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:23.325678 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:23.325838 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:23.336799 (+ 10961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:23.336811 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:23.336832 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:23.336922 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:23.344191 (+ 7269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:23.346530 (+ 2339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:23.346557 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:23.346565 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:23.346911 (+ 346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:23.348617 (+ 1706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:23.348634 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:23.348640 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:23.349345 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:23.349618 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:23.351362 (+ 1744us) server_negotiation.cc:299] Negotiation successful
0806 02:45:23.351886 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":472,"threads_started":1}
I20260806 02:45:23.355654 4034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:23.305322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:40885 (local address 127.1.210.67:45445)
0806 02:45:23.306986 (+ 1664us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:23.307048 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:23.310903 (+ 3855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:23.312757 (+ 1854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:23.312768 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:23.313610 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:23.315905 (+ 2295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:23.315996 (+ 91us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:23.319292 (+ 3296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:23.319305 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:23.324085 (+ 4780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:23.324137 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:23.326183 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:23.326198 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:23.326543 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:23.328042 (+ 1499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:23.328125 (+ 83us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:23.336097 (+ 7972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:23.344628 (+ 8531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:23.344862 (+ 234us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:23.344911 (+ 49us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:23.346262 (+ 1351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:23.347722 (+ 1460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:23.347742 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:23.347749 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:23.348326 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:23.349605 (+ 1279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:23.349633 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:23.350532 (+ 899us) client_negotiation.cc:769] Sending connection context
0806 02:45:23.351102 (+ 570us) client_negotiation.cc:240] Negotiation successful
0806 02:45:23.352110 (+ 1008us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1211,"thread_start_us":763,"threads_started":1}
I20260806 02:45:23.382177 4009 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.67:41745
I20260806 02:45:23.382239 4144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.67:41745 every 8 connection(s)
I20260806 02:45:23.385039 4009 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
I20260806 02:45:23.387025 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 4009
I20260806 02:45:23.387307 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance
I20260806 02:45:23.388423 1865 external_mini_cluster.cc:1428] Setting key 21f6a4e7b3108047cdce6f34b668b133
I20260806 02:45:23.389158 4145 heartbeater.cc:347] Connected to a master server at 127.1.210.126:40885
I20260806 02:45:23.389995 4145 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:23.392138 4145 heartbeater.cc:511] Master 127.1.210.126:40885 requested a full tablet report, sending...
I20260806 02:45:23.394713 3651 ts_manager.cc:194] Registered new tserver with Master: 8af34451093b451097f4352ab317f2ae (127.1.210.67:41745)
I20260806 02:45:23.396346 3651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.67@KRBTEST.COM'} at 127.1.210.67:45445
I20260806 02:45:23.396855 4132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:23.371747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:40885 (local address 127.1.210.67:54591)
0806 02:45:23.373480 (+ 1733us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:23.373526 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:23.373821 (+ 295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:23.374489 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:23.374509 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:23.375507 (+ 998us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:23.376299 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:23.376331 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:23.378366 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:23.378370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:23.380635 (+ 2265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:23.380726 (+ 91us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:23.382518 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:23.382527 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:23.382831 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:23.383809 (+ 978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:23.383837 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:23.384958 (+ 1121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:23.391147 (+ 6189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:23.391156 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:23.391160 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:23.392782 (+ 1622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:23.393782 (+ 1000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:23.393787 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:23.393794 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:23.394086 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:23.395412 (+ 1326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:23.395424 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:23.395709 (+ 285us) client_negotiation.cc:769] Sending connection context
0806 02:45:23.395927 (+ 218us) client_negotiation.cc:240] Negotiation successful
0806 02:45:23.396288 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1387,"thread_start_us":1110,"threads_started":1}
I20260806 02:45:23.397517 4033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:23.371746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:54591 (local address 127.1.210.126:40885)
0806 02:45:23.372094 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:23.372112 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:23.373678 (+ 1566us) server_negotiation.cc:407] Connection header received
0806 02:45:23.374030 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:23.374035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:23.374203 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:23.374384 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:23.376595 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:23.378164 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:23.380977 (+ 2813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:23.382177 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:23.382432 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:23.385255 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:23.385266 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:23.385269 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:23.385317 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:23.390946 (+ 5629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:23.393051 (+ 2105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:23.393065 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:23.393080 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:23.393563 (+ 483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:23.394338 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:23.394344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:23.394347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:23.395148 (+ 801us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:23.395449 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:23.396122 (+ 673us) server_negotiation.cc:299] Negotiation successful
0806 02:45:23.396544 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 02:45:23.408418 1865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:45:23 dist-test-slave-8tdl krb5kdc[3606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 02:45:23 dist-test-slave-8tdl krb5kdc[3606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:23.497351 4033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:23.468879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36436 (local address 127.1.210.126:40885)
0806 02:45:23.469239 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:23.469245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:23.470289 (+ 1044us) server_negotiation.cc:407] Connection header received
0806 02:45:23.470986 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:23.470993 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:23.471076 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:23.471196 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:23.473033 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:23.474101 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:23.476035 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:23.476968 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:23.477145 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:23.485352 (+ 8207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:23.485367 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:23.485371 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:23.485442 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:23.490669 (+ 5227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:23.492462 (+ 1793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:23.492469 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:23.492472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:23.492874 (+ 402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:23.493764 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:23.493782 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:23.493803 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:23.494747 (+ 944us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:23.495349 (+ 602us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:23.496171 (+ 822us) server_negotiation.cc:299] Negotiation successful
0806 02:45:23.496647 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260806 02:45:23.507225 3651 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:45:23.522616 3651 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:45:23.524643 4005 heartbeater.cc:503] Master 127.1.210.126:40885 was elected leader, sending a full tablet report...
I20260806 02:45:23.571753 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:23.548121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33102 (local address 127.1.210.66:35359)
0806 02:45:23.549902 (+ 1781us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:23.549918 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:23.551418 (+ 1500us) server_negotiation.cc:407] Connection header received
0806 02:45:23.552081 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:23.552108 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:23.552632 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:23.553070 (+ 438us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:23.555152 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:23.557771 (+ 2619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:23.563891 (+ 6120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:23.567418 (+ 3527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:23.567813 (+ 395us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:23.568170 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:23.570867 (+ 2697us) server_negotiation.cc:299] Negotiation successful
0806 02:45:23.571282 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":610,"threads_started":1}
I20260806 02:45:23.572556 4158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:23.547847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:35359 (local address 127.0.0.1:33102)
0806 02:45:23.550626 (+ 2779us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:23.550755 (+ 129us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:23.551111 (+ 356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:23.553840 (+ 2729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:23.553875 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:23.553971 (+ 96us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:23.554651 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:23.554740 (+ 89us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:23.558270 (+ 3530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:23.558308 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:23.563547 (+ 5239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:23.563582 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:23.568416 (+ 4834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:23.568428 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:23.569105 (+ 677us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:23.569211 (+ 106us) client_negotiation.cc:769] Sending connection context
0806 02:45:23.569537 (+ 326us) client_negotiation.cc:240] Negotiation successful
0806 02:45:23.570085 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1362,"mutex_wait_us":4,"thread_start_us":559,"threads_started":1}
I20260806 02:45:23.582193 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:23.550561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60218 (local address 127.1.210.65:46461)
0806 02:45:23.566456 (+ 15895us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:23.566486 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:23.566580 (+ 94us) server_negotiation.cc:407] Connection header received
0806 02:45:23.566804 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:23.566832 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:23.567678 (+ 846us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:23.568625 (+ 947us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:23.570627 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:23.573300 (+ 2673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:23.578352 (+ 5052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:23.580584 (+ 2232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:23.580830 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:23.581494 (+ 664us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:23.581607 (+ 113us) server_negotiation.cc:299] Negotiation successful
0806 02:45:23.581841 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15272,"thread_start_us":17344,"threads_started":1}
I20260806 02:45:23.581907 3933 tablet_service.cc:1467] Processing CreateTablet for tablet 3513e1371faa45e4bf2c5ac5a51cfead (DEFAULT_TABLE table=test-table [id=f2f1945182c342f7bdf642e97affb1a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:23.584018 4163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:23.550474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:46461 (local address 127.0.0.1:60218)
0806 02:45:23.563419 (+ 12945us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:23.563453 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:23.563694 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:23.568325 (+ 4631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:23.568333 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:23.568375 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:23.570085 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:23.570118 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:23.573657 (+ 3539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:23.573678 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:23.577980 (+ 4302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:23.578022 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:23.580882 (+ 2860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:23.580890 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:23.581130 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:23.581172 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 02:45:23.581275 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 02:45:23.581440 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12669,"mutex_wait_us":458,"thread_start_us":18713,"threads_started":1}
I20260806 02:45:23.585278 3933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3513e1371faa45e4bf2c5ac5a51cfead. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:23.589912 4161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:23.548913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:41745 (local address 127.0.0.1:53146)
0806 02:45:23.570077 (+ 21164us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:23.570119 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:23.570435 (+ 316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:23.572589 (+ 2154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:23.572598 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:23.572641 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:23.573260 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:23.573276 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:23.579953 (+ 6677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:23.579960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:23.584898 (+ 4938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:23.584920 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:23.588461 (+ 3541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:23.588483 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:23.589131 (+ 648us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:23.589182 (+ 51us) client_negotiation.cc:769] Sending connection context
0806 02:45:23.589306 (+ 124us) client_negotiation.cc:240] Negotiation successful
0806 02:45:23.589479 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20755,"mutex_wait_us":1716,"thread_start_us":12142,"threads_started":1}
I20260806 02:45:23.589951 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:23.549841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53146 (local address 127.1.210.67:41745)
0806 02:45:23.568633 (+ 18792us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:23.568645 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:23.570764 (+ 2119us) server_negotiation.cc:407] Connection header received
0806 02:45:23.570990 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:23.571017 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:23.571446 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:23.571932 (+ 486us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:23.573716 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:23.579569 (+ 5853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:23.585221 (+ 5652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:23.587556 (+ 2335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:23.587715 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:23.588225 (+ 510us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:23.589390 (+ 1165us) server_negotiation.cc:299] Negotiation successful
0806 02:45:23.589595 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18497,"thread_start_us":18760,"threads_started":1}
I20260806 02:45:23.591440 3791 tablet_service.cc:1467] Processing CreateTablet for tablet 3513e1371faa45e4bf2c5ac5a51cfead (DEFAULT_TABLE table=test-table [id=f2f1945182c342f7bdf642e97affb1a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:23.593954 3791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3513e1371faa45e4bf2c5ac5a51cfead. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:23.597277 4075 tablet_service.cc:1467] Processing CreateTablet for tablet 3513e1371faa45e4bf2c5ac5a51cfead (DEFAULT_TABLE table=test-table [id=f2f1945182c342f7bdf642e97affb1a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:23.599082 4075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3513e1371faa45e4bf2c5ac5a51cfead. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:23.612322 4164 tablet_bootstrap.cc:492] T 3513e1371faa45e4bf2c5ac5a51cfead P 87d16739d49043fdb3241c9486c4292e: Bootstrap starting.
I20260806 02:45:23.620066 4164 tablet_bootstrap.cc:654] T 3513e1371faa45e4bf2c5ac5a51cfead P 87d16739d49043fdb3241c9486c4292e: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:23.621320 4165 tablet_bootstrap.cc:492] T 3513e1371faa45e4bf2c5ac5a51cfead P 4c122a1f079c4ce78297526b89f5f60a: Bootstrap starting.
I20260806 02:45:23.623703 4164 log.cc:824] T 3513e1371faa45e4bf2c5ac5a51cfead P 87d16739d49043fdb3241c9486c4292e: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:23.629042 4165 tablet_bootstrap.cc:654] T 3513e1371faa45e4bf2c5ac5a51cfead P 4c122a1f079c4ce78297526b89f5f60a: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:23.630869 4164 tablet_bootstrap.cc:492] T 3513e1371faa45e4bf2c5ac5a51cfead P 87d16739d49043fdb3241c9486c4292e: No bootstrap required, opened a new log
I20260806 02:45:23.630806 4165 log.cc:824] T 3513e1371faa45e4bf2c5ac5a51cfead P 4c122a1f079c4ce78297526b89f5f60a: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:23.631474 4164 ts_tablet_manager.cc:1397] T 3513e1371faa45e4bf2c5ac5a51cfead P 87d16739d49043fdb3241c9486c4292e: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.008s
I20260806 02:45:23.633304 4166 tablet_bootstrap.cc:492] T 3513e1371faa45e4bf2c5ac5a51cfead P 8af34451093b451097f4352ab317f2ae: Bootstrap starting.
I20260806 02:45:23.639674 4165 tablet_bootstrap.cc:492] T 3513e1371faa45e4bf2c5ac5a51cfead P 4c122a1f079c4ce78297526b89f5f60a: No bootstrap required, opened a new log
I20260806 02:45:23.640599 4165 ts_tablet_manager.cc:1397] T 3513e1371faa45e4bf2c5ac5a51cfead P 4c122a1f079c4ce78297526b89f5f60a: Time spent bootstrapping tablet: real 0.020s user 0.017s sys 0.000s
I20260806 02:45:23.641703 4166 tablet_bootstrap.cc:654] T 3513e1371faa45e4bf2c5ac5a51cfead P 8af34451093b451097f4352ab317f2ae: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:23.645140 4166 log.cc:824] T 3513e1371faa45e4bf2c5ac5a51cfead P 8af34451093b451097f4352ab317f2ae: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:23.646575 4164 raft_consensus.cc:348] T 3513e1371faa45e4bf2c5ac5a51cfead P 87d16739d49043fdb3241c9486c4292e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:23.647591 4164 raft_consensus.cc:374] T 3513e1371faa45e4bf2c5ac5a51cfead P 87d16739d49043fdb3241c9486c4292e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:23.647907 4164 raft_consensus.cc:729] T 3513e1371faa45e4bf2c5ac5a51cfead P 87d16739d49043fdb3241c9486c4292e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87d16739d49043fdb3241c9486c4292e, State: Initialized, Role: FOLLOWER
I20260806 02:45:23.649537 4164 consensus_queue.cc:260] T 3513e1371faa45e4bf2c5ac5a51cfead P 87d16739d49043fdb3241c9486c4292e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:23.655637 4166 tablet_bootstrap.cc:492] T 3513e1371faa45e4bf2c5ac5a51cfead P 8af34451093b451097f4352ab317f2ae: No bootstrap required, opened a new log
I20260806 02:45:23.656319 4166 ts_tablet_manager.cc:1397] T 3513e1371faa45e4bf2c5ac5a51cfead P 8af34451093b451097f4352ab317f2ae: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.009s
I20260806 02:45:23.657051 4164 ts_tablet_manager.cc:1428] T 3513e1371faa45e4bf2c5ac5a51cfead P 87d16739d49043fdb3241c9486c4292e: Time spent starting tablet: real 0.025s user 0.015s sys 0.005s
I20260806 02:45:23.661680 4165 raft_consensus.cc:348] T 3513e1371faa45e4bf2c5ac5a51cfead P 4c122a1f079c4ce78297526b89f5f60a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:23.662781 4165 raft_consensus.cc:374] T 3513e1371faa45e4bf2c5ac5a51cfead P 4c122a1f079c4ce78297526b89f5f60a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:23.662994 4165 raft_consensus.cc:729] T 3513e1371faa45e4bf2c5ac5a51cfead P 4c122a1f079c4ce78297526b89f5f60a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c122a1f079c4ce78297526b89f5f60a, State: Initialized, Role: FOLLOWER
I20260806 02:45:23.664469 4165 consensus_queue.cc:260] T 3513e1371faa45e4bf2c5ac5a51cfead P 4c122a1f079c4ce78297526b89f5f60a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:23.668466 4165 ts_tablet_manager.cc:1428] T 3513e1371faa45e4bf2c5ac5a51cfead P 4c122a1f079c4ce78297526b89f5f60a: Time spent starting tablet: real 0.027s user 0.021s sys 0.004s
I20260806 02:45:23.673700 4166 raft_consensus.cc:348] T 3513e1371faa45e4bf2c5ac5a51cfead P 8af34451093b451097f4352ab317f2ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:23.674897 4166 raft_consensus.cc:374] T 3513e1371faa45e4bf2c5ac5a51cfead P 8af34451093b451097f4352ab317f2ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:23.675103 4166 raft_consensus.cc:729] T 3513e1371faa45e4bf2c5ac5a51cfead P 8af34451093b451097f4352ab317f2ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8af34451093b451097f4352ab317f2ae, State: Initialized, Role: FOLLOWER
I20260806 02:45:23.676471 4166 consensus_queue.cc:260] T 3513e1371faa45e4bf2c5ac5a51cfead P 8af34451093b451097f4352ab317f2ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:23.679109 4145 heartbeater.cc:503] Master 127.1.210.126:40885 was elected leader, sending a full tablet report...
I20260806 02:45:23.680851 4166 ts_tablet_manager.cc:1428] T 3513e1371faa45e4bf2c5ac5a51cfead P 8af34451093b451097f4352ab317f2ae: Time spent starting tablet: real 0.024s user 0.021s sys 0.003s
W20260806 02:45:23.746027 3863 tablet.cc:2369] T 3513e1371faa45e4bf2c5ac5a51cfead P 4c122a1f079c4ce78297526b89f5f60a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:45:23.764081 4006 tablet.cc:2369] T 3513e1371faa45e4bf2c5ac5a51cfead P 87d16739d49043fdb3241c9486c4292e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:45:23.849710 4171 raft_consensus.cc:482] T 3513e1371faa45e4bf2c5ac5a51cfead P 4c122a1f079c4ce78297526b89f5f60a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:45:23.850399 4171 raft_consensus.cc:504] T 3513e1371faa45e4bf2c5ac5a51cfead P 4c122a1f079c4ce78297526b89f5f60a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:23.854049 4171 leader_election.cc:288] T 3513e1371faa45e4bf2c5ac5a51cfead P 4c122a1f079c4ce78297526b89f5f60a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87d16739d49043fdb3241c9486c4292e (127.1.210.66:35359), 8af34451093b451097f4352ab317f2ae (127.1.210.67:41745)
I20260806 02:45:23.867458 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:23.854446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:54949 (local address 127.1.210.66:35359)
0806 02:45:23.854925 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:23.854936 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:23.856081 (+ 1145us) server_negotiation.cc:407] Connection header received
0806 02:45:23.856621 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:23.856634 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:23.856790 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:23.856947 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:23.858598 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:23.860194 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:23.864183 (+ 3989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:23.865249 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:23.865496 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:23.865579 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:23.866616 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0806 02:45:23.866903 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241}
I20260806 02:45:23.867516 4173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:23.854392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:35359 (local address 127.1.210.65:54949)
0806 02:45:23.855831 (+ 1439us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:23.855901 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:23.856299 (+ 398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:23.857042 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:23.857061 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:23.857154 (+ 93us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:23.858236 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:23.858252 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:23.860604 (+ 2352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:23.860628 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:23.863771 (+ 3143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:23.863808 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:23.865745 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:23.865767 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:23.866167 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:23.866258 (+ 91us) client_negotiation.cc:769] Sending connection context
0806 02:45:23.866466 (+ 208us) client_negotiation.cc:240] Negotiation successful
0806 02:45:23.866769 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":977,"mutex_wait_us":60,"thread_start_us":1000,"threads_started":1}
I20260806 02:45:23.867978 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:23.854912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:51119 (local address 127.1.210.67:41745)
0806 02:45:23.855128 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:23.855134 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:23.856284 (+ 1150us) server_negotiation.cc:407] Connection header received
0806 02:45:23.856524 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:23.856531 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:23.856675 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:23.856820 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:23.858233 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:23.859861 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:23.863953 (+ 4092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:23.865944 (+ 1991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:23.866235 (+ 291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:23.866371 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:23.867189 (+ 818us) server_negotiation.cc:299] Negotiation successful
0806 02:45:23.867466 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 02:45:23.869028 4174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:23.854475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:41745 (local address 127.1.210.65:51119)
0806 02:45:23.855907 (+ 1432us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:23.855951 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:23.856307 (+ 356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:23.856887 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:23.856901 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:23.856938 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:23.857970 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:23.857984 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:23.860122 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:23.860129 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:23.863612 (+ 3483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:23.863634 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:23.866330 (+ 2696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:23.866336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:23.866798 (+ 462us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:23.866847 (+ 49us) client_negotiation.cc:769] Sending connection context
0806 02:45:23.867146 (+ 299us) client_negotiation.cc:240] Negotiation successful
0806 02:45:23.867464 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1185,"mutex_wait_us":179,"spinlock_wait_cycles":88832,"thread_start_us":489,"threads_started":1}
I20260806 02:45:23.869987 3953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3513e1371faa45e4bf2c5ac5a51cfead" candidate_uuid: "4c122a1f079c4ce78297526b89f5f60a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87d16739d49043fdb3241c9486c4292e" is_pre_election: true
I20260806 02:45:23.871434 4095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3513e1371faa45e4bf2c5ac5a51cfead" candidate_uuid: "4c122a1f079c4ce78297526b89f5f60a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8af34451093b451097f4352ab317f2ae" is_pre_election: true
I20260806 02:45:23.871788 3953 raft_consensus.cc:2454] T 3513e1371faa45e4bf2c5ac5a51cfead P 87d16739d49043fdb3241c9486c4292e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c122a1f079c4ce78297526b89f5f60a in term 0.
I20260806 02:45:23.872390 4095 raft_consensus.cc:2454] T 3513e1371faa45e4bf2c5ac5a51cfead P 8af34451093b451097f4352ab317f2ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c122a1f079c4ce78297526b89f5f60a in term 0.
I20260806 02:45:23.873224 3740 leader_election.cc:302] T 3513e1371faa45e4bf2c5ac5a51cfead P 4c122a1f079c4ce78297526b89f5f60a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c122a1f079c4ce78297526b89f5f60a, 87d16739d49043fdb3241c9486c4292e; no voters:
I20260806 02:45:23.874018 4171 raft_consensus.cc:2781] T 3513e1371faa45e4bf2c5ac5a51cfead P 4c122a1f079c4ce78297526b89f5f60a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:45:23.874383 4171 raft_consensus.cc:482] T 3513e1371faa45e4bf2c5ac5a51cfead P 4c122a1f079c4ce78297526b89f5f60a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:45:23.874748 4171 raft_consensus.cc:3037] T 3513e1371faa45e4bf2c5ac5a51cfead P 4c122a1f079c4ce78297526b89f5f60a [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:23.880093 4171 raft_consensus.cc:504] T 3513e1371faa45e4bf2c5ac5a51cfead P 4c122a1f079c4ce78297526b89f5f60a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:23.881419 4171 leader_election.cc:288] T 3513e1371faa45e4bf2c5ac5a51cfead P 4c122a1f079c4ce78297526b89f5f60a [CANDIDATE]: Term 1 election: Requested vote from peers 87d16739d49043fdb3241c9486c4292e (127.1.210.66:35359), 8af34451093b451097f4352ab317f2ae (127.1.210.67:41745)
I20260806 02:45:23.882416 3953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3513e1371faa45e4bf2c5ac5a51cfead" candidate_uuid: "4c122a1f079c4ce78297526b89f5f60a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87d16739d49043fdb3241c9486c4292e"
I20260806 02:45:23.882905 3953 raft_consensus.cc:3037] T 3513e1371faa45e4bf2c5ac5a51cfead P 87d16739d49043fdb3241c9486c4292e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:23.882668 4095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3513e1371faa45e4bf2c5ac5a51cfead" candidate_uuid: "4c122a1f079c4ce78297526b89f5f60a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8af34451093b451097f4352ab317f2ae"
I20260806 02:45:23.883348 4095 raft_consensus.cc:3037] T 3513e1371faa45e4bf2c5ac5a51cfead P 8af34451093b451097f4352ab317f2ae [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:23.886925 3953 raft_consensus.cc:2454] T 3513e1371faa45e4bf2c5ac5a51cfead P 87d16739d49043fdb3241c9486c4292e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c122a1f079c4ce78297526b89f5f60a in term 1.
I20260806 02:45:23.887866 3740 leader_election.cc:302] T 3513e1371faa45e4bf2c5ac5a51cfead P 4c122a1f079c4ce78297526b89f5f60a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c122a1f079c4ce78297526b89f5f60a, 87d16739d49043fdb3241c9486c4292e; no voters:
I20260806 02:45:23.888167 4095 raft_consensus.cc:2454] T 3513e1371faa45e4bf2c5ac5a51cfead P 8af34451093b451097f4352ab317f2ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c122a1f079c4ce78297526b89f5f60a in term 1.
I20260806 02:45:23.888370 4171 raft_consensus.cc:2781] T 3513e1371faa45e4bf2c5ac5a51cfead P 4c122a1f079c4ce78297526b89f5f60a [term 1 FOLLOWER]: Leader election won for term 1
W20260806 02:45:23.889701 4146 tablet.cc:2369] T 3513e1371faa45e4bf2c5ac5a51cfead P 8af34451093b451097f4352ab317f2ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:45:23.890405 4171 raft_consensus.cc:686] T 3513e1371faa45e4bf2c5ac5a51cfead P 4c122a1f079c4ce78297526b89f5f60a [term 1 LEADER]: Becoming Leader. State: Replica: 4c122a1f079c4ce78297526b89f5f60a, State: Running, Role: LEADER
I20260806 02:45:23.891912 4171 consensus_queue.cc:237] T 3513e1371faa45e4bf2c5ac5a51cfead P 4c122a1f079c4ce78297526b89f5f60a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:23.901667 3653 catalog_manager.cc:5495] T 3513e1371faa45e4bf2c5ac5a51cfead P 4c122a1f079c4ce78297526b89f5f60a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c122a1f079c4ce78297526b89f5f60a (127.1.210.65). New cstate: current_term: 1 leader_uuid: "4c122a1f079c4ce78297526b89f5f60a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } health_report { overall_health: UNKNOWN } } }
Aug 06 02:45:24 dist-test-slave-8tdl krb5kdc[3606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984320, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:24.024681 4183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:23.998827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:40885 (local address 127.0.0.1:36452)
0806 02:45:23.999957 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:23.999985 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:24.000255 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:24.000662 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:24.000667 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:24.001727 (+ 1060us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:24.002142 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:24.002154 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.003961 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.003983 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:24.005523 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:24.005547 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.006780 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.006787 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:24.007238 (+ 451us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:24.008372 (+ 1134us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:24.008404 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:24.012604 (+ 4200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:24.018269 (+ 5665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:24.018286 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:24.018305 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:24.019480 (+ 1175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:24.020119 (+ 639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:24.020124 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:24.020126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:24.020303 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:24.021430 (+ 1127us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:24.021441 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:24.022127 (+ 686us) client_negotiation.cc:769] Sending connection context
0806 02:45:24.022769 (+ 642us) client_negotiation.cc:240] Negotiation successful
0806 02:45:24.023914 (+ 1145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":880,"mutex_wait_us":50,"thread_start_us":580,"threads_started":1}
I20260806 02:45:24.025807 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:23.998827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36452 (local address 127.1.210.126:40885)
0806 02:45:23.999811 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:23.999843 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:24.000122 (+ 279us) server_negotiation.cc:407] Connection header received
0806 02:45:24.000436 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:24.000441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:24.000510 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:24.000637 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:24.002445 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.003772 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.005760 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.006453 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.006760 (+ 307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:24.013039 (+ 6279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:24.013045 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:24.013058 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:24.013109 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:24.018009 (+ 4900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:24.019743 (+ 1734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:24.019748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:24.019751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:24.019946 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:24.020471 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:24.020475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:24.020477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:24.021043 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:24.021659 (+ 616us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:24.024553 (+ 2894us) server_negotiation.cc:299] Negotiation successful
0806 02:45:24.025191 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":768,"thread_start_us":526,"threads_started":1}
I20260806 02:45:24.034955 3653 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:36452:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 02:45:24.037154 3653 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:45:24.052731 3791 tablet_service.cc:1467] Processing CreateTablet for tablet 109a52d41ff541d9b1feb4346bfb373a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0a8c89fc2506449f8fef2635b44c6fc9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:45:24.054454 3933 tablet_service.cc:1467] Processing CreateTablet for tablet 109a52d41ff541d9b1feb4346bfb373a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0a8c89fc2506449f8fef2635b44c6fc9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:45:24.056277 3933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 109a52d41ff541d9b1feb4346bfb373a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:24.056269 3791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 109a52d41ff541d9b1feb4346bfb373a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:24.055262 4075 tablet_service.cc:1467] Processing CreateTablet for tablet 109a52d41ff541d9b1feb4346bfb373a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0a8c89fc2506449f8fef2635b44c6fc9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:45:24.057024 4075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 109a52d41ff541d9b1feb4346bfb373a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:24.069700 4164 tablet_bootstrap.cc:492] T 109a52d41ff541d9b1feb4346bfb373a P 87d16739d49043fdb3241c9486c4292e: Bootstrap starting.
I20260806 02:45:24.069754 4166 tablet_bootstrap.cc:492] T 109a52d41ff541d9b1feb4346bfb373a P 8af34451093b451097f4352ab317f2ae: Bootstrap starting.
I20260806 02:45:24.070125 4165 tablet_bootstrap.cc:492] T 109a52d41ff541d9b1feb4346bfb373a P 4c122a1f079c4ce78297526b89f5f60a: Bootstrap starting.
I20260806 02:45:24.073309 4164 tablet_bootstrap.cc:654] T 109a52d41ff541d9b1feb4346bfb373a P 87d16739d49043fdb3241c9486c4292e: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:24.073848 4165 tablet_bootstrap.cc:654] T 109a52d41ff541d9b1feb4346bfb373a P 4c122a1f079c4ce78297526b89f5f60a: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:24.074218 4166 tablet_bootstrap.cc:654] T 109a52d41ff541d9b1feb4346bfb373a P 8af34451093b451097f4352ab317f2ae: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:24.088536 4165 tablet_bootstrap.cc:492] T 109a52d41ff541d9b1feb4346bfb373a P 4c122a1f079c4ce78297526b89f5f60a: No bootstrap required, opened a new log
I20260806 02:45:24.088989 4164 tablet_bootstrap.cc:492] T 109a52d41ff541d9b1feb4346bfb373a P 87d16739d49043fdb3241c9486c4292e: No bootstrap required, opened a new log
I20260806 02:45:24.088971 4165 ts_tablet_manager.cc:1397] T 109a52d41ff541d9b1feb4346bfb373a P 4c122a1f079c4ce78297526b89f5f60a: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.008s
I20260806 02:45:24.089313 4164 ts_tablet_manager.cc:1397] T 109a52d41ff541d9b1feb4346bfb373a P 87d16739d49043fdb3241c9486c4292e: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.002s
I20260806 02:45:24.090732 4165 raft_consensus.cc:348] T 109a52d41ff541d9b1feb4346bfb373a P 4c122a1f079c4ce78297526b89f5f60a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:24.090849 4164 raft_consensus.cc:348] T 109a52d41ff541d9b1feb4346bfb373a P 87d16739d49043fdb3241c9486c4292e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:24.091261 4165 raft_consensus.cc:374] T 109a52d41ff541d9b1feb4346bfb373a P 4c122a1f079c4ce78297526b89f5f60a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:24.091286 4164 raft_consensus.cc:374] T 109a52d41ff541d9b1feb4346bfb373a P 87d16739d49043fdb3241c9486c4292e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:24.091459 4164 raft_consensus.cc:729] T 109a52d41ff541d9b1feb4346bfb373a P 87d16739d49043fdb3241c9486c4292e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87d16739d49043fdb3241c9486c4292e, State: Initialized, Role: FOLLOWER
I20260806 02:45:24.091475 4165 raft_consensus.cc:729] T 109a52d41ff541d9b1feb4346bfb373a P 4c122a1f079c4ce78297526b89f5f60a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c122a1f079c4ce78297526b89f5f60a, State: Initialized, Role: FOLLOWER
I20260806 02:45:24.091957 4164 consensus_queue.cc:260] T 109a52d41ff541d9b1feb4346bfb373a P 87d16739d49043fdb3241c9486c4292e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:24.092077 4165 consensus_queue.cc:260] T 109a52d41ff541d9b1feb4346bfb373a P 4c122a1f079c4ce78297526b89f5f60a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:24.092294 4166 tablet_bootstrap.cc:492] T 109a52d41ff541d9b1feb4346bfb373a P 8af34451093b451097f4352ab317f2ae: No bootstrap required, opened a new log
I20260806 02:45:24.092558 4166 ts_tablet_manager.cc:1397] T 109a52d41ff541d9b1feb4346bfb373a P 8af34451093b451097f4352ab317f2ae: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.008s
I20260806 02:45:24.092954 4171 tablet_replica.cc:439] T 109a52d41ff541d9b1feb4346bfb373a P 4c122a1f079c4ce78297526b89f5f60a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } } }
I20260806 02:45:24.092934 4170 tablet_replica.cc:439] T 109a52d41ff541d9b1feb4346bfb373a P 87d16739d49043fdb3241c9486c4292e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } } }
I20260806 02:45:24.093778 4165 ts_tablet_manager.cc:1428] T 109a52d41ff541d9b1feb4346bfb373a P 4c122a1f079c4ce78297526b89f5f60a: Time spent starting tablet: real 0.004s user 0.002s sys 0.003s
I20260806 02:45:24.093823 4170 tablet_replica.cc:442] T 109a52d41ff541d9b1feb4346bfb373a P 87d16739d49043fdb3241c9486c4292e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:45:24.093860 4171 tablet_replica.cc:442] T 109a52d41ff541d9b1feb4346bfb373a P 4c122a1f079c4ce78297526b89f5f60a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:45:24.095824 4164 ts_tablet_manager.cc:1428] T 109a52d41ff541d9b1feb4346bfb373a P 87d16739d49043fdb3241c9486c4292e: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260806 02:45:24.096792 4166 raft_consensus.cc:348] T 109a52d41ff541d9b1feb4346bfb373a P 8af34451093b451097f4352ab317f2ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:24.097311 4166 raft_consensus.cc:374] T 109a52d41ff541d9b1feb4346bfb373a P 8af34451093b451097f4352ab317f2ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:24.097508 4166 raft_consensus.cc:729] T 109a52d41ff541d9b1feb4346bfb373a P 8af34451093b451097f4352ab317f2ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8af34451093b451097f4352ab317f2ae, State: Initialized, Role: FOLLOWER
I20260806 02:45:24.098119 4166 consensus_queue.cc:260] T 109a52d41ff541d9b1feb4346bfb373a P 8af34451093b451097f4352ab317f2ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:24.099426 4166 ts_tablet_manager.cc:1428] T 109a52d41ff541d9b1feb4346bfb373a P 8af34451093b451097f4352ab317f2ae: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260806 02:45:24.099200 4172 tablet_replica.cc:439] T 109a52d41ff541d9b1feb4346bfb373a P 8af34451093b451097f4352ab317f2ae: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } } }
I20260806 02:45:24.099877 4172 tablet_replica.cc:442] T 109a52d41ff541d9b1feb4346bfb373a P 8af34451093b451097f4352ab317f2ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:45:24.110759 4170 raft_consensus.cc:482] T 109a52d41ff541d9b1feb4346bfb373a P 87d16739d49043fdb3241c9486c4292e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:45:24.111223 4170 raft_consensus.cc:504] T 109a52d41ff541d9b1feb4346bfb373a P 87d16739d49043fdb3241c9486c4292e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:24.117339 4170 leader_election.cc:288] T 109a52d41ff541d9b1feb4346bfb373a P 87d16739d49043fdb3241c9486c4292e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c122a1f079c4ce78297526b89f5f60a (127.1.210.65:46461), 8af34451093b451097f4352ab317f2ae (127.1.210.67:41745)
I20260806 02:45:24.128523 4188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:24.115068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:50435 (local address 127.1.210.65:46461)
0806 02:45:24.117043 (+ 1975us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:24.117074 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:24.118014 (+ 940us) server_negotiation.cc:407] Connection header received
0806 02:45:24.118101 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:24.118114 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:24.118215 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:24.118399 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:24.119467 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.120670 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.124159 (+ 3489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.126539 (+ 2380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.126879 (+ 340us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:24.127021 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:24.127777 (+ 756us) server_negotiation.cc:299] Negotiation successful
0806 02:45:24.127986 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1583,"thread_start_us":1158,"threads_started":1}
I20260806 02:45:24.128621 4189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:24.114736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:46461 (local address 127.1.210.66:50435)
0806 02:45:24.117387 (+ 2651us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:24.117416 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:24.117794 (+ 378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:24.118444 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:24.118461 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:24.118527 (+ 66us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:24.119210 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:24.119229 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.120901 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.120924 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:24.123830 (+ 2906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:24.123862 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.127004 (+ 3142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.127016 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:24.127408 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:24.127471 (+ 63us) client_negotiation.cc:769] Sending connection context
0806 02:45:24.127676 (+ 205us) client_negotiation.cc:240] Negotiation successful
0806 02:45:24.127931 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2457,"thread_start_us":3369,"threads_started":1}
I20260806 02:45:24.129278 4187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:24.115058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:41745 (local address 127.1.210.66:53115)
0806 02:45:24.118651 (+ 3593us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:24.118749 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:24.119035 (+ 286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:24.119627 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:24.119632 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:24.119672 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:24.120189 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:24.120200 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.121887 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.121897 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:24.124978 (+ 3081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:24.125008 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.127744 (+ 2736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.127752 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:24.128039 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:24.128116 (+ 77us) client_negotiation.cc:769] Sending connection context
0806 02:45:24.128294 (+ 178us) client_negotiation.cc:240] Negotiation successful
0806 02:45:24.128507 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3298,"thread_start_us":1742,"threads_started":1}
I20260806 02:45:24.129899 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:24.115245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:53115 (local address 127.1.210.67:41745)
0806 02:45:24.115769 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:24.115784 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:24.118906 (+ 3122us) server_negotiation.cc:407] Connection header received
0806 02:45:24.119303 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:24.119308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:24.119402 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:24.119577 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:24.120424 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.121642 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.125292 (+ 3650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.127408 (+ 2116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.127672 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:24.127805 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:24.128365 (+ 560us) server_negotiation.cc:299] Negotiation successful
0806 02:45:24.128602 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202}
I20260806 02:45:24.130337 3811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "109a52d41ff541d9b1feb4346bfb373a" candidate_uuid: "87d16739d49043fdb3241c9486c4292e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c122a1f079c4ce78297526b89f5f60a" is_pre_election: true
I20260806 02:45:24.130829 4095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "109a52d41ff541d9b1feb4346bfb373a" candidate_uuid: "87d16739d49043fdb3241c9486c4292e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8af34451093b451097f4352ab317f2ae" is_pre_election: true
I20260806 02:45:24.131127 3811 raft_consensus.cc:2454] T 109a52d41ff541d9b1feb4346bfb373a P 4c122a1f079c4ce78297526b89f5f60a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87d16739d49043fdb3241c9486c4292e in term 0.
I20260806 02:45:24.131315 4095 raft_consensus.cc:2454] T 109a52d41ff541d9b1feb4346bfb373a P 8af34451093b451097f4352ab317f2ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87d16739d49043fdb3241c9486c4292e in term 0.
I20260806 02:45:24.132375 3885 leader_election.cc:302] T 109a52d41ff541d9b1feb4346bfb373a P 87d16739d49043fdb3241c9486c4292e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4c122a1f079c4ce78297526b89f5f60a, 87d16739d49043fdb3241c9486c4292e, 8af34451093b451097f4352ab317f2ae; no voters:
I20260806 02:45:24.133173 4170 raft_consensus.cc:2781] T 109a52d41ff541d9b1feb4346bfb373a P 87d16739d49043fdb3241c9486c4292e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:45:24.133392 4170 raft_consensus.cc:482] T 109a52d41ff541d9b1feb4346bfb373a P 87d16739d49043fdb3241c9486c4292e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:45:24.133607 4170 raft_consensus.cc:3037] T 109a52d41ff541d9b1feb4346bfb373a P 87d16739d49043fdb3241c9486c4292e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:24.138209 4170 raft_consensus.cc:504] T 109a52d41ff541d9b1feb4346bfb373a P 87d16739d49043fdb3241c9486c4292e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:24.139447 4170 leader_election.cc:288] T 109a52d41ff541d9b1feb4346bfb373a P 87d16739d49043fdb3241c9486c4292e [CANDIDATE]: Term 1 election: Requested vote from peers 4c122a1f079c4ce78297526b89f5f60a (127.1.210.65:46461), 8af34451093b451097f4352ab317f2ae (127.1.210.67:41745)
I20260806 02:45:24.140220 3811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "109a52d41ff541d9b1feb4346bfb373a" candidate_uuid: "87d16739d49043fdb3241c9486c4292e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c122a1f079c4ce78297526b89f5f60a"
I20260806 02:45:24.140413 4095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "109a52d41ff541d9b1feb4346bfb373a" candidate_uuid: "87d16739d49043fdb3241c9486c4292e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8af34451093b451097f4352ab317f2ae"
I20260806 02:45:24.140671 3811 raft_consensus.cc:3037] T 109a52d41ff541d9b1feb4346bfb373a P 4c122a1f079c4ce78297526b89f5f60a [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:24.140753 4095 raft_consensus.cc:3037] T 109a52d41ff541d9b1feb4346bfb373a P 8af34451093b451097f4352ab317f2ae [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:24.144881 3811 raft_consensus.cc:2454] T 109a52d41ff541d9b1feb4346bfb373a P 4c122a1f079c4ce78297526b89f5f60a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87d16739d49043fdb3241c9486c4292e in term 1.
I20260806 02:45:24.145699 4095 raft_consensus.cc:2454] T 109a52d41ff541d9b1feb4346bfb373a P 8af34451093b451097f4352ab317f2ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87d16739d49043fdb3241c9486c4292e in term 1.
I20260806 02:45:24.145692 3882 leader_election.cc:302] T 109a52d41ff541d9b1feb4346bfb373a P 87d16739d49043fdb3241c9486c4292e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c122a1f079c4ce78297526b89f5f60a, 87d16739d49043fdb3241c9486c4292e; no voters:
I20260806 02:45:24.146260 4170 raft_consensus.cc:2781] T 109a52d41ff541d9b1feb4346bfb373a P 87d16739d49043fdb3241c9486c4292e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:24.146984 4170 raft_consensus.cc:686] T 109a52d41ff541d9b1feb4346bfb373a P 87d16739d49043fdb3241c9486c4292e [term 1 LEADER]: Becoming Leader. State: Replica: 87d16739d49043fdb3241c9486c4292e, State: Running, Role: LEADER
I20260806 02:45:24.147953 4170 consensus_queue.cc:237] T 109a52d41ff541d9b1feb4346bfb373a P 87d16739d49043fdb3241c9486c4292e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:24.154474 4190 tablet_replica.cc:439] T 109a52d41ff541d9b1feb4346bfb373a P 87d16739d49043fdb3241c9486c4292e: TxnStatusTablet state changed. Reason: New leader 87d16739d49043fdb3241c9486c4292e. Latest consensus state: current_term: 1 leader_uuid: "87d16739d49043fdb3241c9486c4292e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } } }
I20260806 02:45:24.155470 4190 tablet_replica.cc:442] T 109a52d41ff541d9b1feb4346bfb373a P 87d16739d49043fdb3241c9486c4292e: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:45:24.156867 4192 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 02:45:24.157423 4192 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 02:45:24.157794 3650 catalog_manager.cc:5495] T 109a52d41ff541d9b1feb4346bfb373a P 87d16739d49043fdb3241c9486c4292e reported cstate change: term changed from 0 to 1, leader changed from <none> to 87d16739d49043fdb3241c9486c4292e (127.1.210.66). New cstate: current_term: 1 leader_uuid: "87d16739d49043fdb3241c9486c4292e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } health_report { overall_health: UNKNOWN } } }
I20260806 02:45:24.209352 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:24.197325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33118 (local address 127.1.210.66:35359)
0806 02:45:24.197621 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:24.197626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:24.197663 (+ 37us) server_negotiation.cc:407] Connection header received
0806 02:45:24.198127 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:24.198140 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:24.198285 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:24.198480 (+ 195us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:24.199389 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.201545 (+ 2156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.205435 (+ 3890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.205967 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.206171 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:24.207365 (+ 1194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:24.208315 (+ 950us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:24.208489 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:24.208726 (+ 237us) server_negotiation.cc:299] Negotiation successful
0806 02:45:24.208853 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 02:45:24.209649 4183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:24.196901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:35359 (local address 127.0.0.1:33118)
0806 02:45:24.197311 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:24.197358 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:24.197680 (+ 322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:24.198595 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:24.198612 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:24.198641 (+ 29us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:45:24.199102 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:24.199127 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.201810 (+ 2683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.201817 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:24.205102 (+ 3285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:24.205146 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.206268 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.206287 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:24.206712 (+ 425us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:24.206904 (+ 192us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:45:24.208491 (+ 1587us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:45:24.208542 (+ 51us) client_negotiation.cc:769] Sending connection context
0806 02:45:24.208668 (+ 126us) client_negotiation.cc:240] Negotiation successful
0806 02:45:24.209034 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260806 02:45:24.227654 3811 raft_consensus.cc:1261] T 109a52d41ff541d9b1feb4346bfb373a P 4c122a1f079c4ce78297526b89f5f60a [term 1 FOLLOWER]: Refusing update from remote peer 87d16739d49043fdb3241c9486c4292e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:24.227648 4095 raft_consensus.cc:1261] T 109a52d41ff541d9b1feb4346bfb373a P 8af34451093b451097f4352ab317f2ae [term 1 FOLLOWER]: Refusing update from remote peer 87d16739d49043fdb3241c9486c4292e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:24.230096 4170 consensus_queue.cc:1035] T 109a52d41ff541d9b1feb4346bfb373a P 87d16739d49043fdb3241c9486c4292e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:24.230792 4190 consensus_queue.cc:1035] T 109a52d41ff541d9b1feb4346bfb373a P 87d16739d49043fdb3241c9486c4292e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:24.247246 4171 tablet_replica.cc:439] T 109a52d41ff541d9b1feb4346bfb373a P 4c122a1f079c4ce78297526b89f5f60a: TxnStatusTablet state changed. Reason: New leader 87d16739d49043fdb3241c9486c4292e. Latest consensus state: current_term: 1 leader_uuid: "87d16739d49043fdb3241c9486c4292e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } } }
I20260806 02:45:24.248348 4171 tablet_replica.cc:442] T 109a52d41ff541d9b1feb4346bfb373a P 4c122a1f079c4ce78297526b89f5f60a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:45:24.253122 4196 mvcc.cc:205] Tried to move back new op lower bound from 7315391792012578816 to 7315391791717507072. Current Snapshot: MvccSnapshot[applied={T|T < 7315391792012578816}]
I20260806 02:45:24.258036 4172 tablet_replica.cc:439] T 109a52d41ff541d9b1feb4346bfb373a P 8af34451093b451097f4352ab317f2ae: TxnStatusTablet state changed. Reason: New leader 87d16739d49043fdb3241c9486c4292e. Latest consensus state: current_term: 1 leader_uuid: "87d16739d49043fdb3241c9486c4292e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } } }
I20260806 02:45:24.258839 4172 tablet_replica.cc:442] T 109a52d41ff541d9b1feb4346bfb373a P 8af34451093b451097f4352ab317f2ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:45:24.269483 4190 tablet_replica.cc:439] T 109a52d41ff541d9b1feb4346bfb373a P 87d16739d49043fdb3241c9486c4292e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "87d16739d49043fdb3241c9486c4292e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } } }
I20260806 02:45:24.270304 4190 tablet_replica.cc:442] T 109a52d41ff541d9b1feb4346bfb373a P 87d16739d49043fdb3241c9486c4292e: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:45:24.273218 4172 tablet_replica.cc:439] T 109a52d41ff541d9b1feb4346bfb373a P 8af34451093b451097f4352ab317f2ae: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "87d16739d49043fdb3241c9486c4292e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } } }
I20260806 02:45:24.274048 4172 tablet_replica.cc:442] T 109a52d41ff541d9b1feb4346bfb373a P 8af34451093b451097f4352ab317f2ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:45:24.274948 4190 tablet_replica.cc:439] T 109a52d41ff541d9b1feb4346bfb373a P 87d16739d49043fdb3241c9486c4292e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "87d16739d49043fdb3241c9486c4292e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } } }
I20260806 02:45:24.275717 4190 tablet_replica.cc:442] T 109a52d41ff541d9b1feb4346bfb373a P 87d16739d49043fdb3241c9486c4292e: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:45:24.289942 4171 tablet_replica.cc:439] T 109a52d41ff541d9b1feb4346bfb373a P 4c122a1f079c4ce78297526b89f5f60a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "87d16739d49043fdb3241c9486c4292e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } } }
I20260806 02:45:24.290490 4171 tablet_replica.cc:442] T 109a52d41ff541d9b1feb4346bfb373a P 4c122a1f079c4ce78297526b89f5f60a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:45:24.294956 4199 mvcc.cc:205] Tried to move back new op lower bound from 7315391792012578816 to 7315391791717507072. Current Snapshot: MvccSnapshot[applied={T|T < 7315391792012578816}]
I20260806 02:45:24.312371 4188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:24.300407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37018 (local address 127.1.210.65:46461)
0806 02:45:24.300810 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:24.300823 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:24.301587 (+ 764us) server_negotiation.cc:407] Connection header received
0806 02:45:24.302278 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:24.302284 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:24.302431 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:24.302620 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:24.303642 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.304720 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.308460 (+ 3740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.309118 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.309247 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:24.310142 (+ 895us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:24.311424 (+ 1282us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:24.311689 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:24.311764 (+ 75us) server_negotiation.cc:299] Negotiation successful
0806 02:45:24.311855 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260806 02:45:24.343386 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:24.332077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:59093 (local address 127.1.210.66:35359)
0806 02:45:24.332417 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:24.332436 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:24.333105 (+ 669us) server_negotiation.cc:407] Connection header received
0806 02:45:24.333437 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:24.333445 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:24.333640 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:24.333823 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:24.334532 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.336794 (+ 2262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.339436 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.340769 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.341010 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:24.341714 (+ 704us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:24.341984 (+ 270us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:24.342198 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:24.342877 (+ 679us) server_negotiation.cc:299] Negotiation successful
0806 02:45:24.342989 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260806 02:45:24.343612 4208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:24.331868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:35359 (local address 127.1.210.65:59093)
0806 02:45:24.332973 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:24.332993 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:24.333177 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:24.333844 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:24.333848 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:24.333858 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:45:24.334322 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:24.334332 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.337009 (+ 2677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.337014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:24.339230 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:24.339252 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.341064 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.341070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:24.341271 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:24.341392 (+ 121us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:45:24.342317 (+ 925us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:45:24.342343 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 02:45:24.342499 (+ 156us) client_negotiation.cc:240] Negotiation successful
0806 02:45:24.342838 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":848,"thread_start_us":608,"threads_started":1}
I20260806 02:45:24.355163 4095 raft_consensus.cc:1261] T 3513e1371faa45e4bf2c5ac5a51cfead P 8af34451093b451097f4352ab317f2ae [term 1 FOLLOWER]: Refusing update from remote peer 4c122a1f079c4ce78297526b89f5f60a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:24.355665 3953 raft_consensus.cc:1261] T 3513e1371faa45e4bf2c5ac5a51cfead P 87d16739d49043fdb3241c9486c4292e [term 1 FOLLOWER]: Refusing update from remote peer 4c122a1f079c4ce78297526b89f5f60a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:24.356438 4175 consensus_queue.cc:1035] T 3513e1371faa45e4bf2c5ac5a51cfead P 4c122a1f079c4ce78297526b89f5f60a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:24.357152 4171 consensus_queue.cc:1035] T 3513e1371faa45e4bf2c5ac5a51cfead P 4c122a1f079c4ce78297526b89f5f60a [LEADER]: Connected to new peer: Peer: permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:24.362936 4197 mvcc.cc:205] Tried to move back new op lower bound from 7315391792551915520 to 7315391790670266368. Current Snapshot: MvccSnapshot[applied={T|T < 7315391792551915520}]
I20260806 02:45:24.406075 4188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:24.399794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:48661 (local address 127.1.210.65:46461)
0806 02:45:24.399973 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:24.399979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:24.399999 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:45:24.400095 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:24.400102 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:24.400191 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:24.400289 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:24.401114 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.401897 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.403824 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.404509 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.404626 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:24.405283 (+ 657us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:24.405402 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:24.405506 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:24.405673 (+ 167us) server_negotiation.cc:299] Negotiation successful
0806 02:45:24.405754 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:45:24.406448 4213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:24.396843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:46461 (local address 127.1.210.66:48661)
0806 02:45:24.397582 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:24.397606 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:24.397762 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:24.400479 (+ 2717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:24.400484 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:24.400501 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:45:24.400895 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:24.400911 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.402051 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.402054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:24.403662 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:24.403677 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.404830 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.404836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:24.405029 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:24.405103 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:45:24.405508 (+ 405us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:45:24.405522 (+ 14us) client_negotiation.cc:769] Sending connection context
0806 02:45:24.405648 (+ 126us) client_negotiation.cc:240] Negotiation successful
0806 02:45:24.405792 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":4206,"threads_started":1}
I20260806 02:45:24.454162 3650 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:24.454507 3650 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:24.459022 3650 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4934c13314b0457a895cea460da2b23d: Sending DeleteTablet for 3 replicas of tablet 3513e1371faa45e4bf2c5ac5a51cfead
I20260806 02:45:24.460466 4075 tablet_service.cc:1514] Processing DeleteTablet for tablet 3513e1371faa45e4bf2c5ac5a51cfead with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:24 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:53146
I20260806 02:45:24.460496 3933 tablet_service.cc:1514] Processing DeleteTablet for tablet 3513e1371faa45e4bf2c5ac5a51cfead with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:24 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:33102
I20260806 02:45:24.461721 3791 tablet_service.cc:1514] Processing DeleteTablet for tablet 3513e1371faa45e4bf2c5ac5a51cfead with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:24 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:60218
I20260806 02:45:24.463099 3650 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:45:24.463847 3650 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:45:24.474119 3790 tablet_service.cc:1467] Processing CreateTablet for tablet 631f4d91352b48bca7abfcd82d5dacfe (DEFAULT_TABLE table=test-table [id=bbbdaf11d2444cd69ee6188cc26b6feb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:24.474823 3790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 631f4d91352b48bca7abfcd82d5dacfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:24.474741 3932 tablet_service.cc:1467] Processing CreateTablet for tablet 631f4d91352b48bca7abfcd82d5dacfe (DEFAULT_TABLE table=test-table [id=bbbdaf11d2444cd69ee6188cc26b6feb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:24.475445 3932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 631f4d91352b48bca7abfcd82d5dacfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:24.479655 4074 tablet_service.cc:1467] Processing CreateTablet for tablet 631f4d91352b48bca7abfcd82d5dacfe (DEFAULT_TABLE table=test-table [id=bbbdaf11d2444cd69ee6188cc26b6feb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:24.480296 4074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 631f4d91352b48bca7abfcd82d5dacfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:24.484788 4165 tablet_bootstrap.cc:492] T 631f4d91352b48bca7abfcd82d5dacfe P 4c122a1f079c4ce78297526b89f5f60a: Bootstrap starting.
I20260806 02:45:24.486696 4165 tablet_bootstrap.cc:654] T 631f4d91352b48bca7abfcd82d5dacfe P 4c122a1f079c4ce78297526b89f5f60a: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:24.488456 4164 tablet_bootstrap.cc:492] T 631f4d91352b48bca7abfcd82d5dacfe P 87d16739d49043fdb3241c9486c4292e: Bootstrap starting.
I20260806 02:45:24.490263 4164 tablet_bootstrap.cc:654] T 631f4d91352b48bca7abfcd82d5dacfe P 87d16739d49043fdb3241c9486c4292e: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:24.492415 4165 tablet_bootstrap.cc:492] T 631f4d91352b48bca7abfcd82d5dacfe P 4c122a1f079c4ce78297526b89f5f60a: No bootstrap required, opened a new log
I20260806 02:45:24.492607 4165 ts_tablet_manager.cc:1397] T 631f4d91352b48bca7abfcd82d5dacfe P 4c122a1f079c4ce78297526b89f5f60a: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20260806 02:45:24.493461 4165 raft_consensus.cc:348] T 631f4d91352b48bca7abfcd82d5dacfe P 4c122a1f079c4ce78297526b89f5f60a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:24.493736 4165 raft_consensus.cc:374] T 631f4d91352b48bca7abfcd82d5dacfe P 4c122a1f079c4ce78297526b89f5f60a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:24.495075 4165 raft_consensus.cc:729] T 631f4d91352b48bca7abfcd82d5dacfe P 4c122a1f079c4ce78297526b89f5f60a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c122a1f079c4ce78297526b89f5f60a, State: Initialized, Role: FOLLOWER
I20260806 02:45:24.495366 4165 consensus_queue.cc:260] T 631f4d91352b48bca7abfcd82d5dacfe P 4c122a1f079c4ce78297526b89f5f60a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:24.499400 4165 ts_tablet_manager.cc:1428] T 631f4d91352b48bca7abfcd82d5dacfe P 4c122a1f079c4ce78297526b89f5f60a: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20260806 02:45:24.507043 4166 tablet_bootstrap.cc:492] T 631f4d91352b48bca7abfcd82d5dacfe P 8af34451093b451097f4352ab317f2ae: Bootstrap starting.
I20260806 02:45:24.509603 4166 tablet_bootstrap.cc:654] T 631f4d91352b48bca7abfcd82d5dacfe P 8af34451093b451097f4352ab317f2ae: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:24.510329 4218 tablet_replica.cc:330] T 3513e1371faa45e4bf2c5ac5a51cfead P 8af34451093b451097f4352ab317f2ae: stopping tablet replica
I20260806 02:45:24.510851 4218 raft_consensus.cc:2229] T 3513e1371faa45e4bf2c5ac5a51cfead P 8af34451093b451097f4352ab317f2ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:24.511363 4218 raft_consensus.cc:2258] T 3513e1371faa45e4bf2c5ac5a51cfead P 8af34451093b451097f4352ab317f2ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:24.514425 4220 tablet_replica.cc:330] T 3513e1371faa45e4bf2c5ac5a51cfead P 4c122a1f079c4ce78297526b89f5f60a: stopping tablet replica
I20260806 02:45:24.514925 4218 ts_tablet_manager.cc:1905] T 3513e1371faa45e4bf2c5ac5a51cfead P 8af34451093b451097f4352ab317f2ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:24.516862 4220 raft_consensus.cc:2229] T 3513e1371faa45e4bf2c5ac5a51cfead P 4c122a1f079c4ce78297526b89f5f60a [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:45:24.520093 4219 tablet_replica.cc:330] T 3513e1371faa45e4bf2c5ac5a51cfead P 87d16739d49043fdb3241c9486c4292e: stopping tablet replica
I20260806 02:45:24.522158 4166 tablet_bootstrap.cc:492] T 631f4d91352b48bca7abfcd82d5dacfe P 8af34451093b451097f4352ab317f2ae: No bootstrap required, opened a new log
I20260806 02:45:24.522380 4166 ts_tablet_manager.cc:1397] T 631f4d91352b48bca7abfcd82d5dacfe P 8af34451093b451097f4352ab317f2ae: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.004s
I20260806 02:45:24.522871 4219 raft_consensus.cc:2229] T 3513e1371faa45e4bf2c5ac5a51cfead P 87d16739d49043fdb3241c9486c4292e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:24.522173 4220 raft_consensus.cc:2258] T 3513e1371faa45e4bf2c5ac5a51cfead P 4c122a1f079c4ce78297526b89f5f60a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:24.523451 4166 raft_consensus.cc:348] T 631f4d91352b48bca7abfcd82d5dacfe P 8af34451093b451097f4352ab317f2ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:24.523787 4166 raft_consensus.cc:374] T 631f4d91352b48bca7abfcd82d5dacfe P 8af34451093b451097f4352ab317f2ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:24.523905 4166 raft_consensus.cc:729] T 631f4d91352b48bca7abfcd82d5dacfe P 8af34451093b451097f4352ab317f2ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8af34451093b451097f4352ab317f2ae, State: Initialized, Role: FOLLOWER
I20260806 02:45:24.524288 4166 consensus_queue.cc:260] T 631f4d91352b48bca7abfcd82d5dacfe P 8af34451093b451097f4352ab317f2ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:24.524884 4219 raft_consensus.cc:2258] T 3513e1371faa45e4bf2c5ac5a51cfead P 87d16739d49043fdb3241c9486c4292e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:24.521893 4164 tablet_bootstrap.cc:492] T 631f4d91352b48bca7abfcd82d5dacfe P 87d16739d49043fdb3241c9486c4292e: No bootstrap required, opened a new log
I20260806 02:45:24.525384 4164 ts_tablet_manager.cc:1397] T 631f4d91352b48bca7abfcd82d5dacfe P 87d16739d49043fdb3241c9486c4292e: Time spent bootstrapping tablet: real 0.037s user 0.015s sys 0.001s
I20260806 02:45:24.526141 4166 ts_tablet_manager.cc:1428] T 631f4d91352b48bca7abfcd82d5dacfe P 8af34451093b451097f4352ab317f2ae: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260806 02:45:24.529487 4219 ts_tablet_manager.cc:1905] T 3513e1371faa45e4bf2c5ac5a51cfead P 87d16739d49043fdb3241c9486c4292e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:24.530519 4220 ts_tablet_manager.cc:1905] T 3513e1371faa45e4bf2c5ac5a51cfead P 4c122a1f079c4ce78297526b89f5f60a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:24.531061 4218 ts_tablet_manager.cc:1918] T 3513e1371faa45e4bf2c5ac5a51cfead P 8af34451093b451097f4352ab317f2ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:45:24.531311 4218 log.cc:1196] T 3513e1371faa45e4bf2c5ac5a51cfead P 8af34451093b451097f4352ab317f2ae: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-2/wal/wals/3513e1371faa45e4bf2c5ac5a51cfead
I20260806 02:45:24.532016 4218 ts_tablet_manager.cc:1939] T 3513e1371faa45e4bf2c5ac5a51cfead P 8af34451093b451097f4352ab317f2ae: Deleting consensus metadata
I20260806 02:45:24.529933 4164 raft_consensus.cc:348] T 631f4d91352b48bca7abfcd82d5dacfe P 87d16739d49043fdb3241c9486c4292e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:24.532325 4164 raft_consensus.cc:374] T 631f4d91352b48bca7abfcd82d5dacfe P 87d16739d49043fdb3241c9486c4292e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:24.532492 4164 raft_consensus.cc:729] T 631f4d91352b48bca7abfcd82d5dacfe P 87d16739d49043fdb3241c9486c4292e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87d16739d49043fdb3241c9486c4292e, State: Initialized, Role: FOLLOWER
I20260806 02:45:24.532851 4164 consensus_queue.cc:260] T 631f4d91352b48bca7abfcd82d5dacfe P 87d16739d49043fdb3241c9486c4292e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:24.533668 4164 ts_tablet_manager.cc:1428] T 631f4d91352b48bca7abfcd82d5dacfe P 87d16739d49043fdb3241c9486c4292e: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20260806 02:45:24.536923 3638 catalog_manager.cc:4841] TS 8af34451093b451097f4352ab317f2ae (127.1.210.67:41745): tablet 3513e1371faa45e4bf2c5ac5a51cfead (table test-table [id=f2f1945182c342f7bdf642e97affb1a6]) successfully deleted
I20260806 02:45:24.547905 4220 ts_tablet_manager.cc:1918] T 3513e1371faa45e4bf2c5ac5a51cfead P 4c122a1f079c4ce78297526b89f5f60a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:45:24.548260 4220 log.cc:1196] T 3513e1371faa45e4bf2c5ac5a51cfead P 4c122a1f079c4ce78297526b89f5f60a: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-0/wal/wals/3513e1371faa45e4bf2c5ac5a51cfead
I20260806 02:45:24.549036 4220 ts_tablet_manager.cc:1939] T 3513e1371faa45e4bf2c5ac5a51cfead P 4c122a1f079c4ce78297526b89f5f60a: Deleting consensus metadata
I20260806 02:45:24.551224 3635 catalog_manager.cc:4841] TS 4c122a1f079c4ce78297526b89f5f60a (127.1.210.65:46461): tablet 3513e1371faa45e4bf2c5ac5a51cfead (table test-table [id=f2f1945182c342f7bdf642e97affb1a6]) successfully deleted
I20260806 02:45:24.556236 4219 ts_tablet_manager.cc:1918] T 3513e1371faa45e4bf2c5ac5a51cfead P 87d16739d49043fdb3241c9486c4292e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:45:24.556432 4219 log.cc:1196] T 3513e1371faa45e4bf2c5ac5a51cfead P 87d16739d49043fdb3241c9486c4292e: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-1/wal/wals/3513e1371faa45e4bf2c5ac5a51cfead
I20260806 02:45:24.557063 4219 ts_tablet_manager.cc:1939] T 3513e1371faa45e4bf2c5ac5a51cfead P 87d16739d49043fdb3241c9486c4292e: Deleting consensus metadata
I20260806 02:45:24.558977 3635 catalog_manager.cc:4841] TS 87d16739d49043fdb3241c9486c4292e (127.1.210.66:35359): tablet 3513e1371faa45e4bf2c5ac5a51cfead (table test-table [id=f2f1945182c342f7bdf642e97affb1a6]) successfully deleted
I20260806 02:45:24.794430 4172 raft_consensus.cc:482] T 631f4d91352b48bca7abfcd82d5dacfe P 8af34451093b451097f4352ab317f2ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:45:24.794720 4172 raft_consensus.cc:504] T 631f4d91352b48bca7abfcd82d5dacfe P 8af34451093b451097f4352ab317f2ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:24.796939 4172 leader_election.cc:288] T 631f4d91352b48bca7abfcd82d5dacfe P 8af34451093b451097f4352ab317f2ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c122a1f079c4ce78297526b89f5f60a (127.1.210.65:46461), 87d16739d49043fdb3241c9486c4292e (127.1.210.66:35359)
I20260806 02:45:24.806151 4188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:24.797661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:57495 (local address 127.1.210.65:46461)
0806 02:45:24.797946 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:24.797953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:24.797977 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:45:24.798064 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:24.798070 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:24.798150 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:24.798255 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:24.799116 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.800309 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.803114 (+ 2805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.804925 (+ 1811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.805182 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:24.805282 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:24.805602 (+ 320us) server_negotiation.cc:299] Negotiation successful
0806 02:45:24.805732 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260806 02:45:24.806550 4224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:24.797081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:46461 (local address 127.1.210.67:57495)
0806 02:45:24.797747 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:24.797769 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:24.797937 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:24.798436 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:24.798444 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:24.798474 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:24.798857 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:24.798884 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.800454 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.800462 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:24.802859 (+ 2397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:24.802883 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.805202 (+ 2319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.805212 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:24.805410 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:24.805450 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 02:45:24.805572 (+ 122us) client_negotiation.cc:240] Negotiation successful
0806 02:45:24.805716 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":1,"thread_start_us":349,"threads_started":1}
I20260806 02:45:24.806715 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:24.798532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:38715 (local address 127.1.210.66:35359)
0806 02:45:24.798782 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:24.798788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:24.798808 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:45:24.798896 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:24.798914 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:24.799025 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:24.799140 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:24.799848 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.801102 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.803555 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.805127 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.805330 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:24.805404 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:24.806043 (+ 639us) server_negotiation.cc:299] Negotiation successful
0806 02:45:24.806218 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 02:45:24.807693 4211 raft_consensus.cc:482] T 631f4d91352b48bca7abfcd82d5dacfe P 87d16739d49043fdb3241c9486c4292e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:45:24.807624 3811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "631f4d91352b48bca7abfcd82d5dacfe" candidate_uuid: "8af34451093b451097f4352ab317f2ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c122a1f079c4ce78297526b89f5f60a" is_pre_election: true
I20260806 02:45:24.807739 4225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:24.797928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:35359 (local address 127.1.210.67:38715)
0806 02:45:24.798592 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:24.798614 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:24.798747 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:24.799160 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:24.799167 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:24.799197 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:24.799630 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:24.799642 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.801288 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.801294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:24.803406 (+ 2112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:24.803416 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.805558 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.805568 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:24.805807 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:24.805843 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 02:45:24.806985 (+ 1142us) client_negotiation.cc:240] Negotiation successful
0806 02:45:24.807155 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":763,"threads_started":1}
I20260806 02:45:24.807906 3811 raft_consensus.cc:2454] T 631f4d91352b48bca7abfcd82d5dacfe P 4c122a1f079c4ce78297526b89f5f60a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8af34451093b451097f4352ab317f2ae in term 0.
I20260806 02:45:24.807859 4211 raft_consensus.cc:504] T 631f4d91352b48bca7abfcd82d5dacfe P 87d16739d49043fdb3241c9486c4292e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:24.808769 4211 leader_election.cc:288] T 631f4d91352b48bca7abfcd82d5dacfe P 87d16739d49043fdb3241c9486c4292e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c122a1f079c4ce78297526b89f5f60a (127.1.210.65:46461), 8af34451093b451097f4352ab317f2ae (127.1.210.67:41745)
I20260806 02:45:24.808495 3952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "631f4d91352b48bca7abfcd82d5dacfe" candidate_uuid: "8af34451093b451097f4352ab317f2ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87d16739d49043fdb3241c9486c4292e" is_pre_election: true
I20260806 02:45:24.809017 3952 raft_consensus.cc:2454] T 631f4d91352b48bca7abfcd82d5dacfe P 87d16739d49043fdb3241c9486c4292e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8af34451093b451097f4352ab317f2ae in term 0.
I20260806 02:45:24.809085 4025 leader_election.cc:302] T 631f4d91352b48bca7abfcd82d5dacfe P 8af34451093b451097f4352ab317f2ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c122a1f079c4ce78297526b89f5f60a, 8af34451093b451097f4352ab317f2ae; no voters:
I20260806 02:45:24.809430 3811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "631f4d91352b48bca7abfcd82d5dacfe" candidate_uuid: "87d16739d49043fdb3241c9486c4292e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c122a1f079c4ce78297526b89f5f60a" is_pre_election: true
I20260806 02:45:24.809641 4172 raft_consensus.cc:2781] T 631f4d91352b48bca7abfcd82d5dacfe P 8af34451093b451097f4352ab317f2ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:45:24.809753 3811 raft_consensus.cc:2454] T 631f4d91352b48bca7abfcd82d5dacfe P 4c122a1f079c4ce78297526b89f5f60a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87d16739d49043fdb3241c9486c4292e in term 0.
I20260806 02:45:24.809791 4172 raft_consensus.cc:482] T 631f4d91352b48bca7abfcd82d5dacfe P 8af34451093b451097f4352ab317f2ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:45:24.809895 4172 raft_consensus.cc:3037] T 631f4d91352b48bca7abfcd82d5dacfe P 8af34451093b451097f4352ab317f2ae [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:24.810211 3882 leader_election.cc:302] T 631f4d91352b48bca7abfcd82d5dacfe P 87d16739d49043fdb3241c9486c4292e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c122a1f079c4ce78297526b89f5f60a, 87d16739d49043fdb3241c9486c4292e; no voters:
I20260806 02:45:24.810505 4211 raft_consensus.cc:2781] T 631f4d91352b48bca7abfcd82d5dacfe P 87d16739d49043fdb3241c9486c4292e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:45:24.810616 4211 raft_consensus.cc:482] T 631f4d91352b48bca7abfcd82d5dacfe P 87d16739d49043fdb3241c9486c4292e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:45:24.810724 4211 raft_consensus.cc:3037] T 631f4d91352b48bca7abfcd82d5dacfe P 87d16739d49043fdb3241c9486c4292e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:24.810655 4094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "631f4d91352b48bca7abfcd82d5dacfe" candidate_uuid: "87d16739d49043fdb3241c9486c4292e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8af34451093b451097f4352ab317f2ae" is_pre_election: true
I20260806 02:45:24.813319 4172 raft_consensus.cc:504] T 631f4d91352b48bca7abfcd82d5dacfe P 8af34451093b451097f4352ab317f2ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:24.813839 4094 raft_consensus.cc:2379] T 631f4d91352b48bca7abfcd82d5dacfe P 8af34451093b451097f4352ab317f2ae [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 87d16739d49043fdb3241c9486c4292e in current term 1: Already voted for candidate 8af34451093b451097f4352ab317f2ae in this term.
I20260806 02:45:24.813731 4211 raft_consensus.cc:504] T 631f4d91352b48bca7abfcd82d5dacfe P 87d16739d49043fdb3241c9486c4292e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:24.815328 4094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "631f4d91352b48bca7abfcd82d5dacfe" candidate_uuid: "87d16739d49043fdb3241c9486c4292e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8af34451093b451097f4352ab317f2ae"
I20260806 02:45:24.815564 3811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "631f4d91352b48bca7abfcd82d5dacfe" candidate_uuid: "87d16739d49043fdb3241c9486c4292e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c122a1f079c4ce78297526b89f5f60a"
I20260806 02:45:24.815719 4094 raft_consensus.cc:2379] T 631f4d91352b48bca7abfcd82d5dacfe P 8af34451093b451097f4352ab317f2ae [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 87d16739d49043fdb3241c9486c4292e in current term 1: Already voted for candidate 8af34451093b451097f4352ab317f2ae in this term.
I20260806 02:45:24.815927 3811 raft_consensus.cc:3037] T 631f4d91352b48bca7abfcd82d5dacfe P 4c122a1f079c4ce78297526b89f5f60a [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:24.816604 4172 leader_election.cc:288] T 631f4d91352b48bca7abfcd82d5dacfe P 8af34451093b451097f4352ab317f2ae [CANDIDATE]: Term 1 election: Requested vote from peers 4c122a1f079c4ce78297526b89f5f60a (127.1.210.65:46461), 87d16739d49043fdb3241c9486c4292e (127.1.210.66:35359)
I20260806 02:45:24.816881 4211 leader_election.cc:288] T 631f4d91352b48bca7abfcd82d5dacfe P 87d16739d49043fdb3241c9486c4292e [CANDIDATE]: Term 1 election: Requested vote from peers 4c122a1f079c4ce78297526b89f5f60a (127.1.210.65:46461), 8af34451093b451097f4352ab317f2ae (127.1.210.67:41745)
I20260806 02:45:24.817378 3810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "631f4d91352b48bca7abfcd82d5dacfe" candidate_uuid: "8af34451093b451097f4352ab317f2ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c122a1f079c4ce78297526b89f5f60a"
I20260806 02:45:24.817725 3952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "631f4d91352b48bca7abfcd82d5dacfe" candidate_uuid: "8af34451093b451097f4352ab317f2ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87d16739d49043fdb3241c9486c4292e"
I20260806 02:45:24.818216 3952 raft_consensus.cc:2379] T 631f4d91352b48bca7abfcd82d5dacfe P 87d16739d49043fdb3241c9486c4292e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8af34451093b451097f4352ab317f2ae in current term 1: Already voted for candidate 87d16739d49043fdb3241c9486c4292e in this term.
I20260806 02:45:24.819186 4025 leader_election.cc:302] T 631f4d91352b48bca7abfcd82d5dacfe P 8af34451093b451097f4352ab317f2ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8af34451093b451097f4352ab317f2ae; no voters: 4c122a1f079c4ce78297526b89f5f60a, 87d16739d49043fdb3241c9486c4292e
I20260806 02:45:24.819597 4172 raft_consensus.cc:2726] T 631f4d91352b48bca7abfcd82d5dacfe P 8af34451093b451097f4352ab317f2ae [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 02:45:24.821501 3811 raft_consensus.cc:2454] T 631f4d91352b48bca7abfcd82d5dacfe P 4c122a1f079c4ce78297526b89f5f60a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87d16739d49043fdb3241c9486c4292e in term 1.
I20260806 02:45:24.822104 3882 leader_election.cc:302] T 631f4d91352b48bca7abfcd82d5dacfe P 87d16739d49043fdb3241c9486c4292e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c122a1f079c4ce78297526b89f5f60a, 87d16739d49043fdb3241c9486c4292e; no voters: 8af34451093b451097f4352ab317f2ae
I20260806 02:45:24.822751 4211 raft_consensus.cc:2781] T 631f4d91352b48bca7abfcd82d5dacfe P 87d16739d49043fdb3241c9486c4292e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:24.823109 4211 raft_consensus.cc:686] T 631f4d91352b48bca7abfcd82d5dacfe P 87d16739d49043fdb3241c9486c4292e [term 1 LEADER]: Becoming Leader. State: Replica: 87d16739d49043fdb3241c9486c4292e, State: Running, Role: LEADER
I20260806 02:45:24.823486 4211 consensus_queue.cc:237] T 631f4d91352b48bca7abfcd82d5dacfe P 87d16739d49043fdb3241c9486c4292e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:24.827486 3650 catalog_manager.cc:5495] T 631f4d91352b48bca7abfcd82d5dacfe P 87d16739d49043fdb3241c9486c4292e reported cstate change: term changed from 0 to 1, leader changed from <none> to 87d16739d49043fdb3241c9486c4292e (127.1.210.66). New cstate: current_term: 1 leader_uuid: "87d16739d49043fdb3241c9486c4292e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } health_report { overall_health: UNKNOWN } } }
I20260806 02:45:24.882596 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:24.874334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53834 (local address 127.1.210.66:35359)
0806 02:45:24.874562 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:24.874570 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:24.875545 (+ 975us) server_negotiation.cc:407] Connection header received
0806 02:45:24.875700 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:24.875706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:24.875863 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:24.876003 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:24.876649 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.877851 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.879944 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.880772 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.880937 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:24.881540 (+ 603us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:24.881751 (+ 211us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:24.881866 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:24.882078 (+ 212us) server_negotiation.cc:299] Negotiation successful
0806 02:45:24.882195 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 02:45:24.901029 4213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:24.894611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:35359 (local address 127.1.210.66:39543)
0806 02:45:24.894840 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:24.894874 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:24.894969 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:24.895379 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:24.895382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:24.895391 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:45:24.895570 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:24.895577 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.897214 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.897218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:24.898512 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:24.898530 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.899573 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.899581 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:24.899759 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:24.899812 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:45:24.900246 (+ 434us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:45:24.900258 (+ 12us) client_negotiation.cc:769] Sending connection context
0806 02:45:24.900354 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 02:45:24.900483 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260806 02:45:24.901053 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:24.894757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:39543 (local address 127.1.210.66:35359)
0806 02:45:24.895013 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:24.895021 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:24.895051 (+ 30us) server_negotiation.cc:407] Connection header received
0806 02:45:24.895124 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:24.895130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:24.895248 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:24.895377 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:24.895766 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.897028 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.898736 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.899378 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.899526 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:24.900006 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:24.900147 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:24.900246 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:24.900383 (+ 137us) server_negotiation.cc:299] Negotiation successful
0806 02:45:24.900447 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":54}
I20260806 02:45:24.912297 4094 raft_consensus.cc:1261] T 631f4d91352b48bca7abfcd82d5dacfe P 8af34451093b451097f4352ab317f2ae [term 1 FOLLOWER]: Refusing update from remote peer 87d16739d49043fdb3241c9486c4292e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:24.912230 3811 raft_consensus.cc:1261] T 631f4d91352b48bca7abfcd82d5dacfe P 4c122a1f079c4ce78297526b89f5f60a [term 1 FOLLOWER]: Refusing update from remote peer 87d16739d49043fdb3241c9486c4292e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:24.913089 4170 consensus_queue.cc:1035] T 631f4d91352b48bca7abfcd82d5dacfe P 87d16739d49043fdb3241c9486c4292e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:24.913585 4211 consensus_queue.cc:1035] T 631f4d91352b48bca7abfcd82d5dacfe P 87d16739d49043fdb3241c9486c4292e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 02:45:24.945583 4196 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 02:45:26.571012 3650 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:26.571365 3650 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:26.577646 3650 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4934c13314b0457a895cea460da2b23d: Sending DeleteTablet for 3 replicas of tablet 631f4d91352b48bca7abfcd82d5dacfe
I20260806 02:45:26.578783 3791 tablet_service.cc:1514] Processing DeleteTablet for tablet 631f4d91352b48bca7abfcd82d5dacfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:26 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:60218
I20260806 02:45:26.578977 3932 tablet_service.cc:1514] Processing DeleteTablet for tablet 631f4d91352b48bca7abfcd82d5dacfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:26 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:33102
I20260806 02:45:26.578977 4074 tablet_service.cc:1514] Processing DeleteTablet for tablet 631f4d91352b48bca7abfcd82d5dacfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:26 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:53146
I20260806 02:45:26.587610 4255 tablet_replica.cc:330] T 631f4d91352b48bca7abfcd82d5dacfe P 8af34451093b451097f4352ab317f2ae: stopping tablet replica
I20260806 02:45:26.587977 4255 raft_consensus.cc:2229] T 631f4d91352b48bca7abfcd82d5dacfe P 8af34451093b451097f4352ab317f2ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:26.588441 4255 raft_consensus.cc:2258] T 631f4d91352b48bca7abfcd82d5dacfe P 8af34451093b451097f4352ab317f2ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:26.589371 4253 tablet_replica.cc:330] T 631f4d91352b48bca7abfcd82d5dacfe P 4c122a1f079c4ce78297526b89f5f60a: stopping tablet replica
I20260806 02:45:26.589668 4253 raft_consensus.cc:2229] T 631f4d91352b48bca7abfcd82d5dacfe P 4c122a1f079c4ce78297526b89f5f60a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:26.590114 4253 raft_consensus.cc:2258] T 631f4d91352b48bca7abfcd82d5dacfe P 4c122a1f079c4ce78297526b89f5f60a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:26.590298 4255 ts_tablet_manager.cc:1905] T 631f4d91352b48bca7abfcd82d5dacfe P 8af34451093b451097f4352ab317f2ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:26.592306 4253 ts_tablet_manager.cc:1905] T 631f4d91352b48bca7abfcd82d5dacfe P 4c122a1f079c4ce78297526b89f5f60a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:26.597008 4254 tablet_replica.cc:330] T 631f4d91352b48bca7abfcd82d5dacfe P 87d16739d49043fdb3241c9486c4292e: stopping tablet replica
I20260806 02:45:26.597304 4254 raft_consensus.cc:2229] T 631f4d91352b48bca7abfcd82d5dacfe P 87d16739d49043fdb3241c9486c4292e [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:45:26.597978 4254 raft_consensus.cc:2258] T 631f4d91352b48bca7abfcd82d5dacfe P 87d16739d49043fdb3241c9486c4292e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:26.599390 4254 ts_tablet_manager.cc:1905] T 631f4d91352b48bca7abfcd82d5dacfe P 87d16739d49043fdb3241c9486c4292e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:26.600524 4255 ts_tablet_manager.cc:1918] T 631f4d91352b48bca7abfcd82d5dacfe P 8af34451093b451097f4352ab317f2ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 02:45:26.600827 4255 log.cc:1196] T 631f4d91352b48bca7abfcd82d5dacfe P 8af34451093b451097f4352ab317f2ae: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-2/wal/wals/631f4d91352b48bca7abfcd82d5dacfe
I20260806 02:45:26.601558 4253 ts_tablet_manager.cc:1918] T 631f4d91352b48bca7abfcd82d5dacfe P 4c122a1f079c4ce78297526b89f5f60a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 02:45:26.601712 4253 log.cc:1196] T 631f4d91352b48bca7abfcd82d5dacfe P 4c122a1f079c4ce78297526b89f5f60a: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-0/wal/wals/631f4d91352b48bca7abfcd82d5dacfe
I20260806 02:45:26.601704 4255 ts_tablet_manager.cc:1939] T 631f4d91352b48bca7abfcd82d5dacfe P 8af34451093b451097f4352ab317f2ae: Deleting consensus metadata
I20260806 02:45:26.602278 4253 ts_tablet_manager.cc:1939] T 631f4d91352b48bca7abfcd82d5dacfe P 4c122a1f079c4ce78297526b89f5f60a: Deleting consensus metadata
I20260806 02:45:26.603318 3638 catalog_manager.cc:4841] TS 8af34451093b451097f4352ab317f2ae (127.1.210.67:41745): tablet 631f4d91352b48bca7abfcd82d5dacfe (table test-table [id=bbbdaf11d2444cd69ee6188cc26b6feb]) successfully deleted
I20260806 02:45:26.603695 3635 catalog_manager.cc:4841] TS 4c122a1f079c4ce78297526b89f5f60a (127.1.210.65:46461): tablet 631f4d91352b48bca7abfcd82d5dacfe (table test-table [id=bbbdaf11d2444cd69ee6188cc26b6feb]) successfully deleted
I20260806 02:45:26.606351 4254 ts_tablet_manager.cc:1918] T 631f4d91352b48bca7abfcd82d5dacfe P 87d16739d49043fdb3241c9486c4292e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 02:45:26.606511 4254 log.cc:1196] T 631f4d91352b48bca7abfcd82d5dacfe P 87d16739d49043fdb3241c9486c4292e: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-1/wal/wals/631f4d91352b48bca7abfcd82d5dacfe
I20260806 02:45:26.607018 4254 ts_tablet_manager.cc:1939] T 631f4d91352b48bca7abfcd82d5dacfe P 87d16739d49043fdb3241c9486c4292e: Deleting consensus metadata
I20260806 02:45:26.608268 3635 catalog_manager.cc:4841] TS 87d16739d49043fdb3241c9486c4292e (127.1.210.66:35359): tablet 631f4d91352b48bca7abfcd82d5dacfe (table test-table [id=bbbdaf11d2444cd69ee6188cc26b6feb]) successfully deleted
Aug 06 02:45:26 dist-test-slave-8tdl krb5kdc[3606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984326, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 02:45:26 dist-test-slave-8tdl krb5kdc[3606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984326, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:26.657191 4263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:26.628985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58980 (local address 127.1.210.126:40885)
0806 02:45:26.629690 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:26.629695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:26.629717 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:45:26.629776 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:26.629779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:26.629839 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:26.629954 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:26.631506 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:26.632579 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:26.634310 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:26.634822 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:26.634974 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:26.650404 (+ 15430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:26.650411 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:26.650415 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:26.650473 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:26.653674 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:26.654595 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:26.654602 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:26.654606 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:26.654832 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:26.655352 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:26.655359 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:26.655366 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:26.655730 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:26.656034 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:26.656405 (+ 371us) server_negotiation.cc:299] Negotiation successful
0806 02:45:26.656701 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":418,"threads_started":1}
I20260806 02:45:26.661470 3650 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:45:26.662195 3650 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:45:26.669822 3791 tablet_service.cc:1467] Processing CreateTablet for tablet b7028c9754274821a7a6107be7515d4b (DEFAULT_TABLE table=test-table [id=d0e1de93d6af4c9d816914d09c3cc14c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:26.669896 3932 tablet_service.cc:1467] Processing CreateTablet for tablet b7028c9754274821a7a6107be7515d4b (DEFAULT_TABLE table=test-table [id=d0e1de93d6af4c9d816914d09c3cc14c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:26.670055 4074 tablet_service.cc:1467] Processing CreateTablet for tablet b7028c9754274821a7a6107be7515d4b (DEFAULT_TABLE table=test-table [id=d0e1de93d6af4c9d816914d09c3cc14c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:26.670329 3791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7028c9754274821a7a6107be7515d4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:26.670392 3932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7028c9754274821a7a6107be7515d4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:26.670502 4074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7028c9754274821a7a6107be7515d4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:26.686021 4264 tablet_bootstrap.cc:492] T b7028c9754274821a7a6107be7515d4b P 4c122a1f079c4ce78297526b89f5f60a: Bootstrap starting.
I20260806 02:45:26.688583 4265 tablet_bootstrap.cc:492] T b7028c9754274821a7a6107be7515d4b P 8af34451093b451097f4352ab317f2ae: Bootstrap starting.
I20260806 02:45:26.690056 4264 tablet_bootstrap.cc:654] T b7028c9754274821a7a6107be7515d4b P 4c122a1f079c4ce78297526b89f5f60a: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:26.690546 4265 tablet_bootstrap.cc:654] T b7028c9754274821a7a6107be7515d4b P 8af34451093b451097f4352ab317f2ae: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:26.692516 4266 tablet_bootstrap.cc:492] T b7028c9754274821a7a6107be7515d4b P 87d16739d49043fdb3241c9486c4292e: Bootstrap starting.
I20260806 02:45:26.695195 4264 tablet_bootstrap.cc:492] T b7028c9754274821a7a6107be7515d4b P 4c122a1f079c4ce78297526b89f5f60a: No bootstrap required, opened a new log
I20260806 02:45:26.695439 4266 tablet_bootstrap.cc:654] T b7028c9754274821a7a6107be7515d4b P 87d16739d49043fdb3241c9486c4292e: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:26.695484 4264 ts_tablet_manager.cc:1397] T b7028c9754274821a7a6107be7515d4b P 4c122a1f079c4ce78297526b89f5f60a: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260806 02:45:26.695974 4265 tablet_bootstrap.cc:492] T b7028c9754274821a7a6107be7515d4b P 8af34451093b451097f4352ab317f2ae: No bootstrap required, opened a new log
I20260806 02:45:26.696214 4265 ts_tablet_manager.cc:1397] T b7028c9754274821a7a6107be7515d4b P 8af34451093b451097f4352ab317f2ae: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260806 02:45:26.696812 4264 raft_consensus.cc:348] T b7028c9754274821a7a6107be7515d4b P 4c122a1f079c4ce78297526b89f5f60a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:26.697188 4264 raft_consensus.cc:374] T b7028c9754274821a7a6107be7515d4b P 4c122a1f079c4ce78297526b89f5f60a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:26.697319 4264 raft_consensus.cc:729] T b7028c9754274821a7a6107be7515d4b P 4c122a1f079c4ce78297526b89f5f60a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c122a1f079c4ce78297526b89f5f60a, State: Initialized, Role: FOLLOWER
I20260806 02:45:26.697715 4264 consensus_queue.cc:260] T b7028c9754274821a7a6107be7515d4b P 4c122a1f079c4ce78297526b89f5f60a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:26.698735 4264 ts_tablet_manager.cc:1428] T b7028c9754274821a7a6107be7515d4b P 4c122a1f079c4ce78297526b89f5f60a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260806 02:45:26.698982 4265 raft_consensus.cc:348] T b7028c9754274821a7a6107be7515d4b P 8af34451093b451097f4352ab317f2ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:26.699323 4265 raft_consensus.cc:374] T b7028c9754274821a7a6107be7515d4b P 8af34451093b451097f4352ab317f2ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:26.699468 4266 tablet_bootstrap.cc:492] T b7028c9754274821a7a6107be7515d4b P 87d16739d49043fdb3241c9486c4292e: No bootstrap required, opened a new log
I20260806 02:45:26.699528 4265 raft_consensus.cc:729] T b7028c9754274821a7a6107be7515d4b P 8af34451093b451097f4352ab317f2ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8af34451093b451097f4352ab317f2ae, State: Initialized, Role: FOLLOWER
I20260806 02:45:26.699738 4266 ts_tablet_manager.cc:1397] T b7028c9754274821a7a6107be7515d4b P 87d16739d49043fdb3241c9486c4292e: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260806 02:45:26.699915 4265 consensus_queue.cc:260] T b7028c9754274821a7a6107be7515d4b P 8af34451093b451097f4352ab317f2ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:26.701103 4265 ts_tablet_manager.cc:1428] T b7028c9754274821a7a6107be7515d4b P 8af34451093b451097f4352ab317f2ae: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260806 02:45:26.701236 4266 raft_consensus.cc:348] T b7028c9754274821a7a6107be7515d4b P 87d16739d49043fdb3241c9486c4292e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:26.701581 4266 raft_consensus.cc:374] T b7028c9754274821a7a6107be7515d4b P 87d16739d49043fdb3241c9486c4292e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:26.701787 4266 raft_consensus.cc:729] T b7028c9754274821a7a6107be7515d4b P 87d16739d49043fdb3241c9486c4292e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87d16739d49043fdb3241c9486c4292e, State: Initialized, Role: FOLLOWER
I20260806 02:45:26.702150 4266 consensus_queue.cc:260] T b7028c9754274821a7a6107be7515d4b P 87d16739d49043fdb3241c9486c4292e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:26.702852 4266 ts_tablet_manager.cc:1428] T b7028c9754274821a7a6107be7515d4b P 87d16739d49043fdb3241c9486c4292e: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
I20260806 02:45:26.881415 4270 raft_consensus.cc:482] T b7028c9754274821a7a6107be7515d4b P 4c122a1f079c4ce78297526b89f5f60a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:45:26.881654 4270 raft_consensus.cc:504] T b7028c9754274821a7a6107be7515d4b P 4c122a1f079c4ce78297526b89f5f60a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:26.882436 4270 leader_election.cc:288] T b7028c9754274821a7a6107be7515d4b P 4c122a1f079c4ce78297526b89f5f60a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87d16739d49043fdb3241c9486c4292e (127.1.210.66:35359), 8af34451093b451097f4352ab317f2ae (127.1.210.67:41745)
I20260806 02:45:26.883072 3952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7028c9754274821a7a6107be7515d4b" candidate_uuid: "4c122a1f079c4ce78297526b89f5f60a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87d16739d49043fdb3241c9486c4292e" is_pre_election: true
I20260806 02:45:26.883147 4095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7028c9754274821a7a6107be7515d4b" candidate_uuid: "4c122a1f079c4ce78297526b89f5f60a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8af34451093b451097f4352ab317f2ae" is_pre_election: true
I20260806 02:45:26.883370 4095 raft_consensus.cc:2454] T b7028c9754274821a7a6107be7515d4b P 8af34451093b451097f4352ab317f2ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c122a1f079c4ce78297526b89f5f60a in term 0.
I20260806 02:45:26.883348 3952 raft_consensus.cc:2454] T b7028c9754274821a7a6107be7515d4b P 87d16739d49043fdb3241c9486c4292e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c122a1f079c4ce78297526b89f5f60a in term 0.
I20260806 02:45:26.883867 3743 leader_election.cc:302] T b7028c9754274821a7a6107be7515d4b P 4c122a1f079c4ce78297526b89f5f60a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c122a1f079c4ce78297526b89f5f60a, 8af34451093b451097f4352ab317f2ae; no voters:
I20260806 02:45:26.884202 4270 raft_consensus.cc:2781] T b7028c9754274821a7a6107be7515d4b P 4c122a1f079c4ce78297526b89f5f60a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:45:26.884335 4270 raft_consensus.cc:482] T b7028c9754274821a7a6107be7515d4b P 4c122a1f079c4ce78297526b89f5f60a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:45:26.884433 4270 raft_consensus.cc:3037] T b7028c9754274821a7a6107be7515d4b P 4c122a1f079c4ce78297526b89f5f60a [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:26.886482 4270 raft_consensus.cc:504] T b7028c9754274821a7a6107be7515d4b P 4c122a1f079c4ce78297526b89f5f60a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:26.887105 4270 leader_election.cc:288] T b7028c9754274821a7a6107be7515d4b P 4c122a1f079c4ce78297526b89f5f60a [CANDIDATE]: Term 1 election: Requested vote from peers 87d16739d49043fdb3241c9486c4292e (127.1.210.66:35359), 8af34451093b451097f4352ab317f2ae (127.1.210.67:41745)
I20260806 02:45:26.887528 3952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7028c9754274821a7a6107be7515d4b" candidate_uuid: "4c122a1f079c4ce78297526b89f5f60a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87d16739d49043fdb3241c9486c4292e"
I20260806 02:45:26.887800 3952 raft_consensus.cc:3037] T b7028c9754274821a7a6107be7515d4b P 87d16739d49043fdb3241c9486c4292e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:26.887786 4095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7028c9754274821a7a6107be7515d4b" candidate_uuid: "4c122a1f079c4ce78297526b89f5f60a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8af34451093b451097f4352ab317f2ae"
I20260806 02:45:26.888002 4095 raft_consensus.cc:3037] T b7028c9754274821a7a6107be7515d4b P 8af34451093b451097f4352ab317f2ae [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:26.891160 3952 raft_consensus.cc:2454] T b7028c9754274821a7a6107be7515d4b P 87d16739d49043fdb3241c9486c4292e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c122a1f079c4ce78297526b89f5f60a in term 1.
I20260806 02:45:26.891530 4095 raft_consensus.cc:2454] T b7028c9754274821a7a6107be7515d4b P 8af34451093b451097f4352ab317f2ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c122a1f079c4ce78297526b89f5f60a in term 1.
I20260806 02:45:26.891726 3740 leader_election.cc:302] T b7028c9754274821a7a6107be7515d4b P 4c122a1f079c4ce78297526b89f5f60a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c122a1f079c4ce78297526b89f5f60a, 87d16739d49043fdb3241c9486c4292e; no voters:
I20260806 02:45:26.892164 4270 raft_consensus.cc:2781] T b7028c9754274821a7a6107be7515d4b P 4c122a1f079c4ce78297526b89f5f60a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:26.893090 4270 raft_consensus.cc:686] T b7028c9754274821a7a6107be7515d4b P 4c122a1f079c4ce78297526b89f5f60a [term 1 LEADER]: Becoming Leader. State: Replica: 4c122a1f079c4ce78297526b89f5f60a, State: Running, Role: LEADER
I20260806 02:45:26.893494 4270 consensus_queue.cc:237] T b7028c9754274821a7a6107be7515d4b P 4c122a1f079c4ce78297526b89f5f60a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:26.896871 3651 catalog_manager.cc:5495] T b7028c9754274821a7a6107be7515d4b P 4c122a1f079c4ce78297526b89f5f60a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c122a1f079c4ce78297526b89f5f60a (127.1.210.65). New cstate: current_term: 1 leader_uuid: "4c122a1f079c4ce78297526b89f5f60a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } health_report { overall_health: UNKNOWN } } }
I20260806 02:45:26.950197 4275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:26.943457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37024 (local address 127.1.210.65:46461)
0806 02:45:26.944731 (+ 1274us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:26.944736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:26.944772 (+ 36us) server_negotiation.cc:407] Connection header received
0806 02:45:26.944828 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:26.944832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:26.944892 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:26.945081 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:26.945719 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:26.946587 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:26.948226 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:26.948658 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:26.948779 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:26.949319 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:26.949431 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:26.949504 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:26.949710 (+ 206us) server_negotiation.cc:299] Negotiation successful
0806 02:45:26.949821 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1136,"thread_start_us":866,"threads_started":1}
I20260806 02:45:26.960675 4095 raft_consensus.cc:1261] T b7028c9754274821a7a6107be7515d4b P 8af34451093b451097f4352ab317f2ae [term 1 FOLLOWER]: Refusing update from remote peer 4c122a1f079c4ce78297526b89f5f60a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:26.960709 3952 raft_consensus.cc:1261] T b7028c9754274821a7a6107be7515d4b P 87d16739d49043fdb3241c9486c4292e [term 1 FOLLOWER]: Refusing update from remote peer 4c122a1f079c4ce78297526b89f5f60a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:26.961267 4272 consensus_queue.cc:1035] T b7028c9754274821a7a6107be7515d4b P 4c122a1f079c4ce78297526b89f5f60a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:26.961645 4270 consensus_queue.cc:1035] T b7028c9754274821a7a6107be7515d4b P 4c122a1f079c4ce78297526b89f5f60a [LEADER]: Connected to new peer: Peer: permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:27.010773 3651 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:27.011137 3651 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:27.014954 3651 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4934c13314b0457a895cea460da2b23d: Sending DeleteTablet for 3 replicas of tablet b7028c9754274821a7a6107be7515d4b
I20260806 02:45:27.016019 3791 tablet_service.cc:1514] Processing DeleteTablet for tablet b7028c9754274821a7a6107be7515d4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:27 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:60218
I20260806 02:45:27.016108 3932 tablet_service.cc:1514] Processing DeleteTablet for tablet b7028c9754274821a7a6107be7515d4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:27 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:33102
I20260806 02:45:27.016438 4253 tablet_replica.cc:330] T b7028c9754274821a7a6107be7515d4b P 4c122a1f079c4ce78297526b89f5f60a: stopping tablet replica
I20260806 02:45:27.016359 4074 tablet_service.cc:1514] Processing DeleteTablet for tablet b7028c9754274821a7a6107be7515d4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:27 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:53146
I20260806 02:45:27.016444 4254 tablet_replica.cc:330] T b7028c9754274821a7a6107be7515d4b P 87d16739d49043fdb3241c9486c4292e: stopping tablet replica
I20260806 02:45:27.016661 4254 raft_consensus.cc:2229] T b7028c9754274821a7a6107be7515d4b P 87d16739d49043fdb3241c9486c4292e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:27.016690 4253 raft_consensus.cc:2229] T b7028c9754274821a7a6107be7515d4b P 4c122a1f079c4ce78297526b89f5f60a [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:45:27.016919 4254 raft_consensus.cc:2258] T b7028c9754274821a7a6107be7515d4b P 87d16739d49043fdb3241c9486c4292e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:27.017233 4253 raft_consensus.cc:2258] T b7028c9754274821a7a6107be7515d4b P 4c122a1f079c4ce78297526b89f5f60a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:27.017982 3651 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 02:45:27.018528 4253 ts_tablet_manager.cc:1905] T b7028c9754274821a7a6107be7515d4b P 4c122a1f079c4ce78297526b89f5f60a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:27.019424 4255 tablet_replica.cc:330] T b7028c9754274821a7a6107be7515d4b P 8af34451093b451097f4352ab317f2ae: stopping tablet replica
I20260806 02:45:27.019644 4255 raft_consensus.cc:2229] T b7028c9754274821a7a6107be7515d4b P 8af34451093b451097f4352ab317f2ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:27.019850 4255 raft_consensus.cc:2258] T b7028c9754274821a7a6107be7515d4b P 8af34451093b451097f4352ab317f2ae [term 1 FOLLOWER]: Raft consensus is shut down!
W20260806 02:45:27.019030 3651 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:45:27.022940 4255 ts_tablet_manager.cc:1905] T b7028c9754274821a7a6107be7515d4b P 8af34451093b451097f4352ab317f2ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:27.024996 4254 ts_tablet_manager.cc:1905] T b7028c9754274821a7a6107be7515d4b P 87d16739d49043fdb3241c9486c4292e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:27.033349 4074 tablet_service.cc:1467] Processing CreateTablet for tablet 9a73313048654af0ab46e6d9da2b4419 (DEFAULT_TABLE table=test-table [id=1dd547799fe24c2594338e481a2517cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:27.033332 3791 tablet_service.cc:1467] Processing CreateTablet for tablet 9a73313048654af0ab46e6d9da2b4419 (DEFAULT_TABLE table=test-table [id=1dd547799fe24c2594338e481a2517cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:27.033967 4253 ts_tablet_manager.cc:1918] T b7028c9754274821a7a6107be7515d4b P 4c122a1f079c4ce78297526b89f5f60a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:45:27.034044 3791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a73313048654af0ab46e6d9da2b4419. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:27.034106 4255 ts_tablet_manager.cc:1918] T b7028c9754274821a7a6107be7515d4b P 8af34451093b451097f4352ab317f2ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:45:27.034273 4253 log.cc:1196] T b7028c9754274821a7a6107be7515d4b P 4c122a1f079c4ce78297526b89f5f60a: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-0/wal/wals/b7028c9754274821a7a6107be7515d4b
I20260806 02:45:27.034425 4255 log.cc:1196] T b7028c9754274821a7a6107be7515d4b P 8af34451093b451097f4352ab317f2ae: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-2/wal/wals/b7028c9754274821a7a6107be7515d4b
I20260806 02:45:27.034578 4074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a73313048654af0ab46e6d9da2b4419. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:27.035003 4253 ts_tablet_manager.cc:1939] T b7028c9754274821a7a6107be7515d4b P 4c122a1f079c4ce78297526b89f5f60a: Deleting consensus metadata
I20260806 02:45:27.036190 4255 ts_tablet_manager.cc:1939] T b7028c9754274821a7a6107be7515d4b P 8af34451093b451097f4352ab317f2ae: Deleting consensus metadata
I20260806 02:45:27.037719 3638 catalog_manager.cc:4841] TS 8af34451093b451097f4352ab317f2ae (127.1.210.67:41745): tablet b7028c9754274821a7a6107be7515d4b (table test-table [id=d0e1de93d6af4c9d816914d09c3cc14c]) successfully deleted
I20260806 02:45:27.038744 3932 tablet_service.cc:1467] Processing CreateTablet for tablet 9a73313048654af0ab46e6d9da2b4419 (DEFAULT_TABLE table=test-table [id=1dd547799fe24c2594338e481a2517cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:27.039526 3932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a73313048654af0ab46e6d9da2b4419. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:27.040951 3635 catalog_manager.cc:4841] TS 4c122a1f079c4ce78297526b89f5f60a (127.1.210.65:46461): tablet b7028c9754274821a7a6107be7515d4b (table test-table [id=d0e1de93d6af4c9d816914d09c3cc14c]) successfully deleted
I20260806 02:45:27.048264 4266 tablet_bootstrap.cc:492] T 9a73313048654af0ab46e6d9da2b4419 P 87d16739d49043fdb3241c9486c4292e: Bootstrap starting.
I20260806 02:45:27.049710 4264 tablet_bootstrap.cc:492] T 9a73313048654af0ab46e6d9da2b4419 P 4c122a1f079c4ce78297526b89f5f60a: Bootstrap starting.
I20260806 02:45:27.050782 4266 tablet_bootstrap.cc:654] T 9a73313048654af0ab46e6d9da2b4419 P 87d16739d49043fdb3241c9486c4292e: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:27.052112 4254 ts_tablet_manager.cc:1918] T b7028c9754274821a7a6107be7515d4b P 87d16739d49043fdb3241c9486c4292e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:45:27.052372 4264 tablet_bootstrap.cc:654] T 9a73313048654af0ab46e6d9da2b4419 P 4c122a1f079c4ce78297526b89f5f60a: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:27.053112 4254 log.cc:1196] T b7028c9754274821a7a6107be7515d4b P 87d16739d49043fdb3241c9486c4292e: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-1/wal/wals/b7028c9754274821a7a6107be7515d4b
I20260806 02:45:27.053725 4254 ts_tablet_manager.cc:1939] T b7028c9754274821a7a6107be7515d4b P 87d16739d49043fdb3241c9486c4292e: Deleting consensus metadata
I20260806 02:45:27.055294 3635 catalog_manager.cc:4841] TS 87d16739d49043fdb3241c9486c4292e (127.1.210.66:35359): tablet b7028c9754274821a7a6107be7515d4b (table test-table [id=d0e1de93d6af4c9d816914d09c3cc14c]) successfully deleted
I20260806 02:45:27.057682 4264 tablet_bootstrap.cc:492] T 9a73313048654af0ab46e6d9da2b4419 P 4c122a1f079c4ce78297526b89f5f60a: No bootstrap required, opened a new log
I20260806 02:45:27.057945 4264 ts_tablet_manager.cc:1397] T 9a73313048654af0ab46e6d9da2b4419 P 4c122a1f079c4ce78297526b89f5f60a: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260806 02:45:27.058401 4265 tablet_bootstrap.cc:492] T 9a73313048654af0ab46e6d9da2b4419 P 8af34451093b451097f4352ab317f2ae: Bootstrap starting.
I20260806 02:45:27.058960 4264 raft_consensus.cc:348] T 9a73313048654af0ab46e6d9da2b4419 P 4c122a1f079c4ce78297526b89f5f60a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:27.059341 4264 raft_consensus.cc:374] T 9a73313048654af0ab46e6d9da2b4419 P 4c122a1f079c4ce78297526b89f5f60a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:27.059523 4264 raft_consensus.cc:729] T 9a73313048654af0ab46e6d9da2b4419 P 4c122a1f079c4ce78297526b89f5f60a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c122a1f079c4ce78297526b89f5f60a, State: Initialized, Role: FOLLOWER
I20260806 02:45:27.060070 4264 consensus_queue.cc:260] T 9a73313048654af0ab46e6d9da2b4419 P 4c122a1f079c4ce78297526b89f5f60a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:27.061709 4265 tablet_bootstrap.cc:654] T 9a73313048654af0ab46e6d9da2b4419 P 8af34451093b451097f4352ab317f2ae: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:27.062114 4264 ts_tablet_manager.cc:1428] T 9a73313048654af0ab46e6d9da2b4419 P 4c122a1f079c4ce78297526b89f5f60a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260806 02:45:27.064355 4266 tablet_bootstrap.cc:492] T 9a73313048654af0ab46e6d9da2b4419 P 87d16739d49043fdb3241c9486c4292e: No bootstrap required, opened a new log
I20260806 02:45:27.064539 4266 ts_tablet_manager.cc:1397] T 9a73313048654af0ab46e6d9da2b4419 P 87d16739d49043fdb3241c9486c4292e: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.009s
I20260806 02:45:27.065696 4266 raft_consensus.cc:348] T 9a73313048654af0ab46e6d9da2b4419 P 87d16739d49043fdb3241c9486c4292e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:27.066141 4266 raft_consensus.cc:374] T 9a73313048654af0ab46e6d9da2b4419 P 87d16739d49043fdb3241c9486c4292e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:27.066298 4266 raft_consensus.cc:729] T 9a73313048654af0ab46e6d9da2b4419 P 87d16739d49043fdb3241c9486c4292e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87d16739d49043fdb3241c9486c4292e, State: Initialized, Role: FOLLOWER
I20260806 02:45:27.066798 4266 consensus_queue.cc:260] T 9a73313048654af0ab46e6d9da2b4419 P 87d16739d49043fdb3241c9486c4292e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:27.068599 4266 ts_tablet_manager.cc:1428] T 9a73313048654af0ab46e6d9da2b4419 P 87d16739d49043fdb3241c9486c4292e: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260806 02:45:27.072890 4265 tablet_bootstrap.cc:492] T 9a73313048654af0ab46e6d9da2b4419 P 8af34451093b451097f4352ab317f2ae: No bootstrap required, opened a new log
I20260806 02:45:27.073045 4265 ts_tablet_manager.cc:1397] T 9a73313048654af0ab46e6d9da2b4419 P 8af34451093b451097f4352ab317f2ae: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260806 02:45:27.073895 4265 raft_consensus.cc:348] T 9a73313048654af0ab46e6d9da2b4419 P 8af34451093b451097f4352ab317f2ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:27.074260 4265 raft_consensus.cc:374] T 9a73313048654af0ab46e6d9da2b4419 P 8af34451093b451097f4352ab317f2ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:27.074432 4265 raft_consensus.cc:729] T 9a73313048654af0ab46e6d9da2b4419 P 8af34451093b451097f4352ab317f2ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8af34451093b451097f4352ab317f2ae, State: Initialized, Role: FOLLOWER
I20260806 02:45:27.074905 4265 consensus_queue.cc:260] T 9a73313048654af0ab46e6d9da2b4419 P 8af34451093b451097f4352ab317f2ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:27.075681 4265 ts_tablet_manager.cc:1428] T 9a73313048654af0ab46e6d9da2b4419 P 8af34451093b451097f4352ab317f2ae: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260806 02:45:27.300935 4270 raft_consensus.cc:482] T 9a73313048654af0ab46e6d9da2b4419 P 4c122a1f079c4ce78297526b89f5f60a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:45:27.301178 4270 raft_consensus.cc:504] T 9a73313048654af0ab46e6d9da2b4419 P 4c122a1f079c4ce78297526b89f5f60a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:27.302268 4270 leader_election.cc:288] T 9a73313048654af0ab46e6d9da2b4419 P 4c122a1f079c4ce78297526b89f5f60a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87d16739d49043fdb3241c9486c4292e (127.1.210.66:35359), 8af34451093b451097f4352ab317f2ae (127.1.210.67:41745)
I20260806 02:45:27.303046 3952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a73313048654af0ab46e6d9da2b4419" candidate_uuid: "4c122a1f079c4ce78297526b89f5f60a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87d16739d49043fdb3241c9486c4292e" is_pre_election: true
I20260806 02:45:27.303262 4095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a73313048654af0ab46e6d9da2b4419" candidate_uuid: "4c122a1f079c4ce78297526b89f5f60a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8af34451093b451097f4352ab317f2ae" is_pre_election: true
I20260806 02:45:27.303567 4095 raft_consensus.cc:2454] T 9a73313048654af0ab46e6d9da2b4419 P 8af34451093b451097f4352ab317f2ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c122a1f079c4ce78297526b89f5f60a in term 0.
I20260806 02:45:27.303592 3952 raft_consensus.cc:2454] T 9a73313048654af0ab46e6d9da2b4419 P 87d16739d49043fdb3241c9486c4292e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c122a1f079c4ce78297526b89f5f60a in term 0.
I20260806 02:45:27.304207 3743 leader_election.cc:302] T 9a73313048654af0ab46e6d9da2b4419 P 4c122a1f079c4ce78297526b89f5f60a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c122a1f079c4ce78297526b89f5f60a, 8af34451093b451097f4352ab317f2ae; no voters:
I20260806 02:45:27.304641 4270 raft_consensus.cc:2781] T 9a73313048654af0ab46e6d9da2b4419 P 4c122a1f079c4ce78297526b89f5f60a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:45:27.304800 4270 raft_consensus.cc:482] T 9a73313048654af0ab46e6d9da2b4419 P 4c122a1f079c4ce78297526b89f5f60a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:45:27.304910 4270 raft_consensus.cc:3037] T 9a73313048654af0ab46e6d9da2b4419 P 4c122a1f079c4ce78297526b89f5f60a [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:27.309839 4270 raft_consensus.cc:504] T 9a73313048654af0ab46e6d9da2b4419 P 4c122a1f079c4ce78297526b89f5f60a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:27.311040 4270 leader_election.cc:288] T 9a73313048654af0ab46e6d9da2b4419 P 4c122a1f079c4ce78297526b89f5f60a [CANDIDATE]: Term 1 election: Requested vote from peers 87d16739d49043fdb3241c9486c4292e (127.1.210.66:35359), 8af34451093b451097f4352ab317f2ae (127.1.210.67:41745)
I20260806 02:45:27.311826 4095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a73313048654af0ab46e6d9da2b4419" candidate_uuid: "4c122a1f079c4ce78297526b89f5f60a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8af34451093b451097f4352ab317f2ae"
I20260806 02:45:27.311872 3952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a73313048654af0ab46e6d9da2b4419" candidate_uuid: "4c122a1f079c4ce78297526b89f5f60a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87d16739d49043fdb3241c9486c4292e"
I20260806 02:45:27.312233 3952 raft_consensus.cc:3037] T 9a73313048654af0ab46e6d9da2b4419 P 87d16739d49043fdb3241c9486c4292e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:27.312222 4095 raft_consensus.cc:3037] T 9a73313048654af0ab46e6d9da2b4419 P 8af34451093b451097f4352ab317f2ae [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:27.315297 4095 raft_consensus.cc:2454] T 9a73313048654af0ab46e6d9da2b4419 P 8af34451093b451097f4352ab317f2ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c122a1f079c4ce78297526b89f5f60a in term 1.
I20260806 02:45:27.315886 3743 leader_election.cc:302] T 9a73313048654af0ab46e6d9da2b4419 P 4c122a1f079c4ce78297526b89f5f60a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c122a1f079c4ce78297526b89f5f60a, 8af34451093b451097f4352ab317f2ae; no voters:
I20260806 02:45:27.316452 4270 raft_consensus.cc:2781] T 9a73313048654af0ab46e6d9da2b4419 P 4c122a1f079c4ce78297526b89f5f60a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:27.316767 4270 raft_consensus.cc:686] T 9a73313048654af0ab46e6d9da2b4419 P 4c122a1f079c4ce78297526b89f5f60a [term 1 LEADER]: Becoming Leader. State: Replica: 4c122a1f079c4ce78297526b89f5f60a, State: Running, Role: LEADER
I20260806 02:45:27.317145 3952 raft_consensus.cc:2454] T 9a73313048654af0ab46e6d9da2b4419 P 87d16739d49043fdb3241c9486c4292e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c122a1f079c4ce78297526b89f5f60a in term 1.
I20260806 02:45:27.317174 4270 consensus_queue.cc:237] T 9a73313048654af0ab46e6d9da2b4419 P 4c122a1f079c4ce78297526b89f5f60a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } }
I20260806 02:45:27.320590 3653 catalog_manager.cc:5495] T 9a73313048654af0ab46e6d9da2b4419 P 4c122a1f079c4ce78297526b89f5f60a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c122a1f079c4ce78297526b89f5f60a (127.1.210.65). New cstate: current_term: 1 leader_uuid: "4c122a1f079c4ce78297526b89f5f60a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c122a1f079c4ce78297526b89f5f60a" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 } health_report { overall_health: UNKNOWN } } }
I20260806 02:45:27.368214 4095 raft_consensus.cc:1261] T 9a73313048654af0ab46e6d9da2b4419 P 8af34451093b451097f4352ab317f2ae [term 1 FOLLOWER]: Refusing update from remote peer 4c122a1f079c4ce78297526b89f5f60a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:27.368969 3952 raft_consensus.cc:1261] T 9a73313048654af0ab46e6d9da2b4419 P 87d16739d49043fdb3241c9486c4292e [term 1 FOLLOWER]: Refusing update from remote peer 4c122a1f079c4ce78297526b89f5f60a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:27.369174 4270 consensus_queue.cc:1035] T 9a73313048654af0ab46e6d9da2b4419 P 4c122a1f079c4ce78297526b89f5f60a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8af34451093b451097f4352ab317f2ae" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:27.369719 4272 consensus_queue.cc:1035] T 9a73313048654af0ab46e6d9da2b4419 P 4c122a1f079c4ce78297526b89f5f60a [LEADER]: Connected to new peer: Peer: permanent_uuid: "87d16739d49043fdb3241c9486c4292e" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 35359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 02:45:27.413120 4238 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 02:45:29.041774 3651 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:29.042117 3651 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:29.047672 3651 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4934c13314b0457a895cea460da2b23d: Sending DeleteTablet for 3 replicas of tablet 9a73313048654af0ab46e6d9da2b4419
I20260806 02:45:29.048813 3932 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a73313048654af0ab46e6d9da2b4419 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:29 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:33102
I20260806 02:45:29.048913 4074 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a73313048654af0ab46e6d9da2b4419 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:29 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:53146
I20260806 02:45:29.049042 3791 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a73313048654af0ab46e6d9da2b4419 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:29 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:60218
I20260806 02:45:29.049768 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 3724
I20260806 02:45:29.060672 4312 tablet_replica.cc:330] T 9a73313048654af0ab46e6d9da2b4419 P 4c122a1f079c4ce78297526b89f5f60a: stopping tablet replica
I20260806 02:45:29.061180 4312 raft_consensus.cc:2229] T 9a73313048654af0ab46e6d9da2b4419 P 4c122a1f079c4ce78297526b89f5f60a [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:45:29.061899 4312 raft_consensus.cc:2258] T 9a73313048654af0ab46e6d9da2b4419 P 4c122a1f079c4ce78297526b89f5f60a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:29.062795 4310 tablet_replica.cc:330] T 9a73313048654af0ab46e6d9da2b4419 P 87d16739d49043fdb3241c9486c4292e: stopping tablet replica
I20260806 02:45:29.063110 4310 raft_consensus.cc:2229] T 9a73313048654af0ab46e6d9da2b4419 P 87d16739d49043fdb3241c9486c4292e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:29.063478 4310 raft_consensus.cc:2258] T 9a73313048654af0ab46e6d9da2b4419 P 87d16739d49043fdb3241c9486c4292e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:29.063556 4311 tablet_replica.cc:330] T 9a73313048654af0ab46e6d9da2b4419 P 8af34451093b451097f4352ab317f2ae: stopping tablet replica
I20260806 02:45:29.063750 4311 raft_consensus.cc:2229] T 9a73313048654af0ab46e6d9da2b4419 P 8af34451093b451097f4352ab317f2ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:29.064040 4311 raft_consensus.cc:2258] T 9a73313048654af0ab46e6d9da2b4419 P 8af34451093b451097f4352ab317f2ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:29.064270 4312 ts_tablet_manager.cc:1905] T 9a73313048654af0ab46e6d9da2b4419 P 4c122a1f079c4ce78297526b89f5f60a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:29.064898 4310 ts_tablet_manager.cc:1905] T 9a73313048654af0ab46e6d9da2b4419 P 87d16739d49043fdb3241c9486c4292e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:29.065572 4311 ts_tablet_manager.cc:1905] T 9a73313048654af0ab46e6d9da2b4419 P 8af34451093b451097f4352ab317f2ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:29.073710 4310 ts_tablet_manager.cc:1918] T 9a73313048654af0ab46e6d9da2b4419 P 87d16739d49043fdb3241c9486c4292e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 02:45:29.073727 4312 ts_tablet_manager.cc:1918] T 9a73313048654af0ab46e6d9da2b4419 P 4c122a1f079c4ce78297526b89f5f60a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 02:45:29.073972 4310 log.cc:1196] T 9a73313048654af0ab46e6d9da2b4419 P 87d16739d49043fdb3241c9486c4292e: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-1/wal/wals/9a73313048654af0ab46e6d9da2b4419
I20260806 02:45:29.073930 4312 log.cc:1196] T 9a73313048654af0ab46e6d9da2b4419 P 4c122a1f079c4ce78297526b89f5f60a: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-0/wal/wals/9a73313048654af0ab46e6d9da2b4419
I20260806 02:45:29.074324 4311 ts_tablet_manager.cc:1918] T 9a73313048654af0ab46e6d9da2b4419 P 8af34451093b451097f4352ab317f2ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 02:45:29.074600 4312 ts_tablet_manager.cc:1939] T 9a73313048654af0ab46e6d9da2b4419 P 4c122a1f079c4ce78297526b89f5f60a: Deleting consensus metadata
I20260806 02:45:29.074786 4310 ts_tablet_manager.cc:1939] T 9a73313048654af0ab46e6d9da2b4419 P 87d16739d49043fdb3241c9486c4292e: Deleting consensus metadata
I20260806 02:45:29.074782 4311 log.cc:1196] T 9a73313048654af0ab46e6d9da2b4419 P 8af34451093b451097f4352ab317f2ae: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984304136196-1865-0/minicluster-data/ts-2/wal/wals/9a73313048654af0ab46e6d9da2b4419
I20260806 02:45:29.075785 4311 ts_tablet_manager.cc:1939] T 9a73313048654af0ab46e6d9da2b4419 P 8af34451093b451097f4352ab317f2ae: Deleting consensus metadata
I20260806 02:45:29.076089 3635 catalog_manager.cc:4841] TS 87d16739d49043fdb3241c9486c4292e (127.1.210.66:35359): tablet 9a73313048654af0ab46e6d9da2b4419 (table test-table [id=1dd547799fe24c2594338e481a2517cc]) successfully deleted
I20260806 02:45:29.076411 3635 catalog_manager.cc:4841] TS 4c122a1f079c4ce78297526b89f5f60a (127.1.210.65:46461): tablet 9a73313048654af0ab46e6d9da2b4419 (table test-table [id=1dd547799fe24c2594338e481a2517cc]) successfully deleted
I20260806 02:45:29.077338 3638 catalog_manager.cc:4841] TS 8af34451093b451097f4352ab317f2ae (127.1.210.67:41745): tablet 9a73313048654af0ab46e6d9da2b4419 (table test-table [id=1dd547799fe24c2594338e481a2517cc]) successfully deleted
Aug 06 02:45:29 dist-test-slave-8tdl krb5kdc[3606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984326, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.210.65@KRBTEST.COM
I20260806 02:45:29.085671 4313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:29.051448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37032 (local address 127.1.210.65:46461)
0806 02:45:29.062550 (+ 11102us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:29.062558 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:29.062590 (+ 32us) server_negotiation.cc:407] Connection header received
0806 02:45:29.062658 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:29.062663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:29.062787 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:29.062920 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:29.065442 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:29.067196 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:29.069592 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:29.070937 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:29.076032 (+ 5095us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:29.079088 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:29.079103 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:29.079120 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:29.079153 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:29.082545 (+ 3392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:29.083425 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:29.083433 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:29.083435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:29.083585 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:29.084050 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:29.084055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:29.084057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:29.084522 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:29.084775 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:29.085043 (+ 268us) server_negotiation.cc:299] Negotiation successful
0806 02:45:29.085227 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10915,"thread_start_us":10752,"threads_started":1}
W20260806 02:45:29.086431 3857 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37032
W20260806 02:45:29.087322 1865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 3724: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:45:29.087620 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 3724
I20260806 02:45:29.113608 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 3866
Aug 06 02:45:29 dist-test-slave-8tdl krb5kdc[3606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984326, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.210.66@KRBTEST.COM
I20260806 02:45:29.131381 4315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:29.114561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53848 (local address 127.1.210.66:35359)
0806 02:45:29.115185 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:29.115189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:29.115206 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:45:29.115261 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:29.115264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:29.115324 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:29.115451 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:29.116843 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:29.117706 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:29.119174 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:29.119715 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:29.119840 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:29.124457 (+ 4617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:29.124476 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:29.124493 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:29.124539 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:29.128053 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:29.128842 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:29.128850 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:29.128852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:29.129059 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:29.129505 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:29.129508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:29.129510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:29.129889 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:29.130200 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:29.130608 (+ 408us) server_negotiation.cc:299] Negotiation successful
0806 02:45:29.130988 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":382,"threads_started":1}
W20260806 02:45:29.132148 4000 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53848
W20260806 02:45:29.132882 1865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 3866: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:45:29.133050 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 3866
I20260806 02:45:29.160979 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 4009
Aug 06 02:45:29 dist-test-slave-8tdl krb5kdc[3606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984326, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.210.67@KRBTEST.COM
I20260806 02:45:29.178532 4316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:29.161957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39110 (local address 127.1.210.67:41745)
0806 02:45:29.162511 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:29.162516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:29.162534 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:45:29.162587 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:29.162591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:29.162659 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:29.162818 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:29.164190 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:29.165013 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:29.166312 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:29.166792 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:29.166922 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:29.171639 (+ 4717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:29.171657 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:29.171674 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:29.171712 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:29.175276 (+ 3564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:29.176103 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:29.176118 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:29.176121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:29.176280 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:29.176699 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:29.176706 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:29.176709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:29.177138 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:29.177453 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:29.177876 (+ 423us) server_negotiation.cc:299] Negotiation successful
0806 02:45:29.178145 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":311,"threads_started":1}
W20260806 02:45:29.179472 4140 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39110
W20260806 02:45:29.180130 1865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 4009: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:45:29.180373 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 4009
I20260806 02:45:29.200337 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 3619
I20260806 02:45:29.201825 3693 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:45:29.335703 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 3619
2026-08-06T02:45:29Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (10372 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:45:29 dist-test-slave-8tdl krb5kdc[4320](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:45:29 dist-test-slave-8tdl krb5kdc[4320](info): set up 2 sockets
Aug 06 02:45:29 dist-test-slave-8tdl krb5kdc[4320](info): commencing operation
krb5kdc: starting...
W20260806 02:45:30.446395 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:45:30 dist-test-slave-8tdl krb5kdc[4320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:45:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
I20260806 02:45:30.667119 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:45647
--webserver_interface=127.1.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:42325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:45647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:31.014566 4333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:31.014954 4333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:31.015126 4333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:31.022626 4333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:31.022773 4333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:31.022872 4333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:31.022961 4333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:31.023036 4333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:45:31.033221 4333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:42325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:45647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:45647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:31.035020 4333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:31.036444 4333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:31.048940 4338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:31.049024 4339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:31.049001 4341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:31.050197 4333 server_base.cc:1034] running on GCE node
I20260806 02:45:31.051169 4333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:31.052707 4333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:31.053944 4333 hybrid_clock.cc:648] HybridClock initialized: now 1785984331053868 us; error 72 us; skew 500 ppm
I20260806 02:45:31.054323 4333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:45:31 dist-test-slave-8tdl krb5kdc[4320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984331, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:31.057806 4333 init.cc:377] Logged in from keytab as kudu/127.1.210.126@KRBTEST.COM (short username kudu)
I20260806 02:45:31.059508 4333 webserver.cc:460] Webserver started at http://127.1.210.126:33769/ using document root <none> and password file <none>
I20260806 02:45:31.060245 4333 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:31.060396 4333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:31.060729 4333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:31.063932 4333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "5b74280dfbee44b9bd7156d296317cf7"
format_stamp: "Formatted at 2026-08-06 02:45:31 on dist-test-slave-8tdl"
server_key: "2dea70ea88fc928cf00627cebd602f9b"
server_key_iv: "e7947c466d52afd106882603f4a337e9"
server_key_version: "encryptionkey@0"
I20260806 02:45:31.064756 4333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "5b74280dfbee44b9bd7156d296317cf7"
format_stamp: "Formatted at 2026-08-06 02:45:31 on dist-test-slave-8tdl"
server_key: "2dea70ea88fc928cf00627cebd602f9b"
server_key_iv: "e7947c466d52afd106882603f4a337e9"
server_key_version: "encryptionkey@0"
I20260806 02:45:31.070142 4333 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 02:45:31.073674 4348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:31.075101 4333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:31.075289 4333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "5b74280dfbee44b9bd7156d296317cf7"
format_stamp: "Formatted at 2026-08-06 02:45:31 on dist-test-slave-8tdl"
server_key: "2dea70ea88fc928cf00627cebd602f9b"
server_key_iv: "e7947c466d52afd106882603f4a337e9"
server_key_version: "encryptionkey@0"
I20260806 02:45:31.075490 4333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:31.116014 4333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:31.120919 4333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:31.121268 4333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:31.142393 4333 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:45647
I20260806 02:45:31.142436 4399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:45647 every 8 connection(s)
I20260806 02:45:31.143894 4333 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:45:31.148018 4400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:31.148867 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 4333
I20260806 02:45:31.149121 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:45:31.150100 1865 external_mini_cluster.cc:1428] Setting key 07c05ac0a2d6b8a6da2c0de4974a05b1
I20260806 02:45:31.163854 4400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b74280dfbee44b9bd7156d296317cf7: Bootstrap starting.
Aug 06 02:45:31 dist-test-slave-8tdl krb5kdc[4320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:31.167821 4400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b74280dfbee44b9bd7156d296317cf7: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:31.169010 4400 log.cc:824] T 00000000000000000000000000000000 P 5b74280dfbee44b9bd7156d296317cf7: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:31.172518 4400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b74280dfbee44b9bd7156d296317cf7: No bootstrap required, opened a new log
I20260806 02:45:31.176577 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:31.152453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38436 (local address 127.1.210.126:45647)
0806 02:45:31.153357 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:31.153370 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:31.153416 (+ 46us) server_negotiation.cc:407] Connection header received
0806 02:45:31.155259 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:31.155289 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:31.155729 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:31.156088 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:31.158137 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:31.159687 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:31.161628 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:31.162432 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:31.162585 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:31.167559 (+ 4974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:31.167568 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:31.167604 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:31.167652 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:31.171465 (+ 3813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:31.172294 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:31.172304 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:31.172313 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:31.172543 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:31.173063 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:31.173074 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:31.173079 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:31.173818 (+ 739us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:31.174131 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:31.174589 (+ 458us) server_negotiation.cc:299] Negotiation successful
0806 02:45:31.175142 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":436,"threads_started":1}
I20260806 02:45:31.181900 4400 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b74280dfbee44b9bd7156d296317cf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b74280dfbee44b9bd7156d296317cf7" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45647 } }
I20260806 02:45:31.182274 4400 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b74280dfbee44b9bd7156d296317cf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:31.182353 4400 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b74280dfbee44b9bd7156d296317cf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b74280dfbee44b9bd7156d296317cf7, State: Initialized, Role: FOLLOWER
I20260806 02:45:31.182926 4400 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b74280dfbee44b9bd7156d296317cf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b74280dfbee44b9bd7156d296317cf7" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45647 } }
I20260806 02:45:31.183138 4400 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b74280dfbee44b9bd7156d296317cf7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:45:31.183341 4400 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b74280dfbee44b9bd7156d296317cf7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:45:31.183542 4400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b74280dfbee44b9bd7156d296317cf7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:31.185901 4400 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b74280dfbee44b9bd7156d296317cf7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b74280dfbee44b9bd7156d296317cf7" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45647 } }
I20260806 02:45:31.186445 4400 leader_election.cc:302] T 00000000000000000000000000000000 P 5b74280dfbee44b9bd7156d296317cf7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b74280dfbee44b9bd7156d296317cf7; no voters:
I20260806 02:45:31.187115 4400 leader_election.cc:288] T 00000000000000000000000000000000 P 5b74280dfbee44b9bd7156d296317cf7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:45:31.187278 4405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b74280dfbee44b9bd7156d296317cf7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:31.187988 4405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b74280dfbee44b9bd7156d296317cf7 [term 1 LEADER]: Becoming Leader. State: Replica: 5b74280dfbee44b9bd7156d296317cf7, State: Running, Role: LEADER
I20260806 02:45:31.188584 4405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b74280dfbee44b9bd7156d296317cf7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b74280dfbee44b9bd7156d296317cf7" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45647 } }
I20260806 02:45:31.189340 4400 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b74280dfbee44b9bd7156d296317cf7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:31.191838 4407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b74280dfbee44b9bd7156d296317cf7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b74280dfbee44b9bd7156d296317cf7. Latest consensus state: current_term: 1 leader_uuid: "5b74280dfbee44b9bd7156d296317cf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b74280dfbee44b9bd7156d296317cf7" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45647 } } }
I20260806 02:45:31.192524 4407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b74280dfbee44b9bd7156d296317cf7 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:31.192615 4406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b74280dfbee44b9bd7156d296317cf7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b74280dfbee44b9bd7156d296317cf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b74280dfbee44b9bd7156d296317cf7" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45647 } } }
I20260806 02:45:31.193424 4406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b74280dfbee44b9bd7156d296317cf7 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:31.200505 4413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:31.205901 4413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:45:31.216171 4413 catalog_manager.cc:1347] Generated new cluster ID: 4f2dbde0c27f4bb884a4275e76e2ab4b
I20260806 02:45:31.216320 4413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:31.234097 4413 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:45:31.235601 4413 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:31.250124 4413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b74280dfbee44b9bd7156d296317cf7: Generated new TSK 0
I20260806 02:45:31.251197 4413 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab.
Entry for principal kudu/127.1.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab.
I20260806 02:45:31.319881 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.65:0
--local_ip_for_outbound_sockets=127.1.210.65
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:45647
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:42325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:31.673888 4426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:31.674206 4426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:31.674309 4426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:31.682168 4426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:31.682308 4426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:31.682428 4426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.65
I20260806 02:45:31.693116 4426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:42325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:45647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.65
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:31.694496 4426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:31.696023 4426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:31.709733 4432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:31.709789 4434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:31.709997 4431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:31.712105 4426 server_base.cc:1034] running on GCE node
I20260806 02:45:31.712735 4426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:31.713826 4426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:31.715044 4426 hybrid_clock.cc:648] HybridClock initialized: now 1785984331715000 us; error 43 us; skew 500 ppm
I20260806 02:45:31.715392 4426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:45:31 dist-test-slave-8tdl krb5kdc[4320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984331, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:31.727595 4426 init.cc:377] Logged in from keytab as kudu/127.1.210.65@KRBTEST.COM (short username kudu)
I20260806 02:45:31.729803 4426 webserver.cc:460] Webserver started at http://127.1.210.65:34743/ using document root <none> and password file <none>
I20260806 02:45:31.730579 4426 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:31.730749 4426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:31.731077 4426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:31.734108 4426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-0/data/instance:
uuid: "27730b163d6044ce910c0f29a4ea80b9"
format_stamp: "Formatted at 2026-08-06 02:45:31 on dist-test-slave-8tdl"
server_key: "65692e7cae2e0f45b6808f9863cb1481"
server_key_iv: "e036854e8dc4da5882316fda8840faf7"
server_key_version: "encryptionkey@0"
I20260806 02:45:31.735033 4426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance:
uuid: "27730b163d6044ce910c0f29a4ea80b9"
format_stamp: "Formatted at 2026-08-06 02:45:31 on dist-test-slave-8tdl"
server_key: "65692e7cae2e0f45b6808f9863cb1481"
server_key_iv: "e036854e8dc4da5882316fda8840faf7"
server_key_version: "encryptionkey@0"
I20260806 02:45:31.740625 4426 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 02:45:31.744526 4441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:31.746058 4426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:45:31.746255 4426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-0/wal
uuid: "27730b163d6044ce910c0f29a4ea80b9"
format_stamp: "Formatted at 2026-08-06 02:45:31 on dist-test-slave-8tdl"
server_key: "65692e7cae2e0f45b6808f9863cb1481"
server_key_iv: "e036854e8dc4da5882316fda8840faf7"
server_key_version: "encryptionkey@0"
I20260806 02:45:31.746502 4426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:31.759855 4426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:31.764720 4426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:31.765108 4426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:31.766145 4426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:31.767983 4426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:31.768097 4426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:31.768244 4426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:31.768316 4426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:31.804572 4426 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.65:34483
I20260806 02:45:31.804623 4554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.65:34483 every 8 connection(s)
I20260806 02:45:31.805935 4426 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
I20260806 02:45:31.815389 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 4426
I20260806 02:45:31.815634 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance
I20260806 02:45:31.816296 1865 external_mini_cluster.cc:1428] Setting key 4f4304568404256f9caaa5b249e13eab
Aug 06 02:45:31 dist-test-slave-8tdl krb5kdc[4320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984331, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:31.828233 4557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:31.808723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:59815 (local address 127.1.210.126:45647)
0806 02:45:31.809183 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:31.809187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:31.810257 (+ 1070us) server_negotiation.cc:407] Connection header received
0806 02:45:31.812262 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:31.812268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:31.812367 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:31.812501 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:31.814091 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:31.815029 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:31.817011 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:31.817452 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:31.817603 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:31.821762 (+ 4159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:31.821767 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:31.821771 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:31.821808 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:31.824728 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:31.825569 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:31.825573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:31.825576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:31.825696 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:31.826095 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:31.826103 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:31.826110 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:31.826501 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:31.826660 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:31.827461 (+ 801us) server_negotiation.cc:299] Negotiation successful
0806 02:45:31.827731 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":233,"threads_started":1}
I20260806 02:45:31.828974 4558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:31.809307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45647 (local address 127.1.210.65:59815)
0806 02:45:31.810059 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:31.810107 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:31.811942 (+ 1835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:31.812743 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:31.812752 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:31.813283 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:31.813898 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:31.813916 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:31.815257 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:31.815261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:31.816806 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:31.816827 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:31.817715 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:31.817721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:31.817942 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:31.818818 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:31.818842 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:31.821494 (+ 2652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:31.824897 (+ 3403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:31.824905 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:31.824919 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:31.825427 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:31.825797 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:31.825800 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:31.825802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:31.825986 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:31.826631 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:31.826637 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:31.827054 (+ 417us) client_negotiation.cc:769] Sending connection context
0806 02:45:31.827308 (+ 254us) client_negotiation.cc:240] Negotiation successful
0806 02:45:31.827715 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":288,"threads_started":1}
I20260806 02:45:31.830741 4555 heartbeater.cc:347] Connected to a master server at 127.1.210.126:45647
I20260806 02:45:31.831069 4555 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:31.831785 4555 heartbeater.cc:511] Master 127.1.210.126:45647 requested a full tablet report, sending...
I20260806 02:45:31.834281 4365 ts_manager.cc:194] Registered new tserver with Master: 27730b163d6044ce910c0f29a4ea80b9 (127.1.210.65:34483)
I20260806 02:45:31.836529 4365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.65@KRBTEST.COM'} at 127.1.210.65:59815
WARNING: no policy specified for kudu/127.1.210.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab.
Entry for principal kudu/127.1.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab.
I20260806 02:45:31.868523 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.66:0
--local_ip_for_outbound_sockets=127.1.210.66
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:45647
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:42325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:32.214462 4561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:32.214880 4561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:32.215088 4561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:32.223104 4561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:32.223227 4561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:32.223413 4561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.66
I20260806 02:45:32.233533 4561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:42325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:45647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.66
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:32.235010 4561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:32.236377 4561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:32.250952 4567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:32.250936 4566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:32.251616 4569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:32.251945 4561 server_base.cc:1034] running on GCE node
I20260806 02:45:32.252573 4561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:32.253620 4561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:32.254835 4561 hybrid_clock.cc:648] HybridClock initialized: now 1785984332254784 us; error 48 us; skew 500 ppm
I20260806 02:45:32.255188 4561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:45:32 dist-test-slave-8tdl krb5kdc[4320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984332, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:32.258405 4561 init.cc:377] Logged in from keytab as kudu/127.1.210.66@KRBTEST.COM (short username kudu)
I20260806 02:45:32.260090 4561 webserver.cc:460] Webserver started at http://127.1.210.66:41915/ using document root <none> and password file <none>
I20260806 02:45:32.260794 4561 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:32.260897 4561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:32.261217 4561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:32.264266 4561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-1/data/instance:
uuid: "6a7c40edccc547f1b6a5f84720c50f08"
format_stamp: "Formatted at 2026-08-06 02:45:32 on dist-test-slave-8tdl"
server_key: "61ad5f8428407deb4bdd7e27caaad891"
server_key_iv: "8f6b41e9a1b4d4610d466910457c54cf"
server_key_version: "encryptionkey@0"
I20260806 02:45:32.265050 4561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance:
uuid: "6a7c40edccc547f1b6a5f84720c50f08"
format_stamp: "Formatted at 2026-08-06 02:45:32 on dist-test-slave-8tdl"
server_key: "61ad5f8428407deb4bdd7e27caaad891"
server_key_iv: "8f6b41e9a1b4d4610d466910457c54cf"
server_key_version: "encryptionkey@0"
I20260806 02:45:32.270530 4561 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260806 02:45:32.274183 4576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:32.275727 4561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:32.275873 4561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-1/wal
uuid: "6a7c40edccc547f1b6a5f84720c50f08"
format_stamp: "Formatted at 2026-08-06 02:45:32 on dist-test-slave-8tdl"
server_key: "61ad5f8428407deb4bdd7e27caaad891"
server_key_iv: "8f6b41e9a1b4d4610d466910457c54cf"
server_key_version: "encryptionkey@0"
I20260806 02:45:32.276050 4561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:32.297518 4561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:32.302500 4561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:32.302848 4561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:32.303889 4561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:32.305307 4561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:32.305387 4561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:32.305562 4561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:32.305614 4561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:32.338006 4561 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.66:39529
I20260806 02:45:32.338089 4690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.66:39529 every 8 connection(s)
I20260806 02:45:32.339464 4561 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
I20260806 02:45:32.340929 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 4561
I20260806 02:45:32.341162 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance
I20260806 02:45:32.341748 1865 external_mini_cluster.cc:1428] Setting key 4b8775ae026a57c161f7540de080f2bb
Aug 06 02:45:32 dist-test-slave-8tdl krb5kdc[4320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984332, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.210.67@KRBTEST.COM; defaulting to no policy
I20260806 02:45:32.362438 4693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:32.342566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:40733 (local address 127.1.210.126:45647)
0806 02:45:32.343051 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:32.343056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:32.344031 (+ 975us) server_negotiation.cc:407] Connection header received
0806 02:45:32.346111 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:32.346116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:32.346215 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:32.346340 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:32.347995 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:32.348710 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:32.350645 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:32.351347 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:32.351463 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:32.356110 (+ 4647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:32.356116 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:32.356119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:32.356157 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:32.359130 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:32.359948 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:32.359952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:32.359955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:32.360092 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:32.360576 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:32.360580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:32.360582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:32.360856 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:32.361028 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:32.361789 (+ 761us) server_negotiation.cc:299] Negotiation successful
0806 02:45:32.362033 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":275,"threads_started":1}
I20260806 02:45:32.363857 4694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:32.343197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45647 (local address 127.1.210.66:40733)
0806 02:45:32.343872 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:32.343908 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:32.345820 (+ 1912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:32.346601 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:32.346610 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:32.347210 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:32.347801 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:32.347819 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:32.348882 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:32.348887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:32.350396 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:32.350415 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:32.351597 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:32.351607 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:32.351951 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:32.352747 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:32.352783 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:32.355811 (+ 3028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:32.359293 (+ 3482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:32.359302 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:32.359321 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:32.359796 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:32.360217 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:32.360222 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:32.360226 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:32.360429 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:32.360991 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:32.361000 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:32.361371 (+ 371us) client_negotiation.cc:769] Sending connection context
0806 02:45:32.361668 (+ 297us) client_negotiation.cc:240] Negotiation successful
0806 02:45:32.362237 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":812,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.67@KRBTEST.COM" created.
I20260806 02:45:32.365440 4691 heartbeater.cc:347] Connected to a master server at 127.1.210.126:45647
I20260806 02:45:32.365784 4691 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:32.366514 4691 heartbeater.cc:511] Master 127.1.210.126:45647 requested a full tablet report, sending...
I20260806 02:45:32.368058 4365 ts_manager.cc:194] Registered new tserver with Master: 6a7c40edccc547f1b6a5f84720c50f08 (127.1.210.66:39529)
I20260806 02:45:32.368928 4365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.66@KRBTEST.COM'} at 127.1.210.66:40733
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab.
Entry for principal kudu/127.1.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab.
I20260806 02:45:32.391633 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.67:0
--local_ip_for_outbound_sockets=127.1.210.67
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:45647
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:42325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:32.735498 4697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:32.735796 4697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:32.735960 4697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:32.743523 4697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:32.743675 4697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:32.743891 4697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.67
I20260806 02:45:32.754221 4697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:42325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:45647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.67
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:32.755650 4697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:32.757023 4697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:32.770639 4703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:32.770747 4702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:32.770780 4705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:32.771657 4697 server_base.cc:1034] running on GCE node
I20260806 02:45:32.772271 4697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:32.773237 4697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:32.774524 4697 hybrid_clock.cc:648] HybridClock initialized: now 1785984332774449 us; error 67 us; skew 500 ppm
I20260806 02:45:32.774940 4697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:45:32 dist-test-slave-8tdl krb5kdc[4320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984332, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:32.778384 4697 init.cc:377] Logged in from keytab as kudu/127.1.210.67@KRBTEST.COM (short username kudu)
I20260806 02:45:32.780395 4697 webserver.cc:460] Webserver started at http://127.1.210.67:40853/ using document root <none> and password file <none>
I20260806 02:45:32.781210 4697 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:32.781354 4697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:32.781685 4697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:32.784765 4697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-2/data/instance:
uuid: "ce5d56b649934d4d824166db0adacca0"
format_stamp: "Formatted at 2026-08-06 02:45:32 on dist-test-slave-8tdl"
server_key: "fc6d7f459cf8f30e8e3966d039470b70"
server_key_iv: "c95cc0605c2b7d27f2d58e54c79510cc"
server_key_version: "encryptionkey@0"
I20260806 02:45:32.785625 4697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance:
uuid: "ce5d56b649934d4d824166db0adacca0"
format_stamp: "Formatted at 2026-08-06 02:45:32 on dist-test-slave-8tdl"
server_key: "fc6d7f459cf8f30e8e3966d039470b70"
server_key_iv: "c95cc0605c2b7d27f2d58e54c79510cc"
server_key_version: "encryptionkey@0"
I20260806 02:45:32.791370 4697 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 02:45:32.795007 4712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:32.796576 4697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:45:32.796769 4697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-2/wal
uuid: "ce5d56b649934d4d824166db0adacca0"
format_stamp: "Formatted at 2026-08-06 02:45:32 on dist-test-slave-8tdl"
server_key: "fc6d7f459cf8f30e8e3966d039470b70"
server_key_iv: "c95cc0605c2b7d27f2d58e54c79510cc"
server_key_version: "encryptionkey@0"
I20260806 02:45:32.796984 4697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:32.827744 4697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:32.832438 4697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:32.832760 4697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:32.833748 4697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:32.835210 4697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:32.835318 4697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:32.835474 4697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:32.835564 4697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:32.839951 4555 heartbeater.cc:503] Master 127.1.210.126:45647 was elected leader, sending a full tablet report...
I20260806 02:45:32.870591 4697 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.67:41899
I20260806 02:45:32.870633 4824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.67:41899 every 8 connection(s)
I20260806 02:45:32.872006 4697 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
I20260806 02:45:32.873118 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 4697
I20260806 02:45:32.873345 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance
I20260806 02:45:32.873888 1865 external_mini_cluster.cc:1428] Setting key d647556fb6d2d924a4134cfa136d215a
Aug 06 02:45:32 dist-test-slave-8tdl krb5kdc[4320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984332, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:32.894033 4827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:32.875174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:44533 (local address 127.1.210.126:45647)
0806 02:45:32.875645 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:32.875651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:32.876716 (+ 1065us) server_negotiation.cc:407] Connection header received
0806 02:45:32.878790 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:32.878794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:32.878862 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:32.878985 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:32.880771 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:32.881542 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:32.883197 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:32.883603 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:32.883741 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:32.887371 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:32.887377 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:32.887380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:32.887416 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:32.890783 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:32.891614 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:32.891620 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:32.891623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:32.891756 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:32.892203 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:32.892208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:32.892218 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:32.892495 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:32.892665 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:32.893414 (+ 749us) server_negotiation.cc:299] Negotiation successful
0806 02:45:32.893642 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":223,"threads_started":1}
I20260806 02:45:32.895135 4828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:32.875738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45647 (local address 127.1.210.67:44533)
0806 02:45:32.876550 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:32.876604 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:32.878469 (+ 1865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:32.879314 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:32.879323 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:32.879857 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:32.880596 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:32.880615 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:32.881708 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:32.881719 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:32.883040 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:32.883053 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:32.883839 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:32.883846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:32.884102 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:32.884776 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:32.884799 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:32.887123 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:32.890971 (+ 3848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:32.890986 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:32.891001 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:32.891479 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:32.891871 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:32.891875 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:32.891879 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:32.892080 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:32.892640 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:32.892648 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:32.893019 (+ 371us) client_negotiation.cc:769] Sending connection context
0806 02:45:32.893299 (+ 280us) client_negotiation.cc:240] Negotiation successful
0806 02:45:32.893875 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":360,"threads_started":1}
I20260806 02:45:32.896510 4825 heartbeater.cc:347] Connected to a master server at 127.1.210.126:45647
I20260806 02:45:32.896793 4825 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:32.897454 4825 heartbeater.cc:511] Master 127.1.210.126:45647 requested a full tablet report, sending...
I20260806 02:45:32.898873 4365 ts_manager.cc:194] Registered new tserver with Master: ce5d56b649934d4d824166db0adacca0 (127.1.210.67:41899)
I20260806 02:45:32.899654 4365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.67@KRBTEST.COM'} at 127.1.210.67:44533
I20260806 02:45:32.902798 1865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 02:45:32.936612 4827 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:32.934028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38452 (local address 127.1.210.126:45647)
0806 02:45:32.934254 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:32.934263 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:32.934652 (+ 389us) server_negotiation.cc:407] Connection header received
0806 02:45:32.934918 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:32.934922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:32.934991 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:32.935071 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:32.936327 (+ 1256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38452: BlockingRecv error: recv got EOF from 127.0.0.1:38452 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 02:45:32.940235 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 4426
W20260806 02:45:32.943269 1865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 4426: Not authorized: Client connection negotiation failed: client connection to 127.1.210.65:34483: server requires authentication, but client does not have Kerberos credentials available
I20260806 02:45:32.943470 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 4426
I20260806 02:45:32.959628 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 4561
W20260806 02:45:32.962590 1865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 4561: Not authorized: Client connection negotiation failed: client connection to 127.1.210.66:39529: server requires authentication, but client does not have Kerberos credentials available
I20260806 02:45:32.962805 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 4561
I20260806 02:45:32.978860 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 4697
W20260806 02:45:32.981966 1865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 4697: Not authorized: Client connection negotiation failed: client connection to 127.1.210.67:41899: server requires authentication, but client does not have Kerberos credentials available
I20260806 02:45:32.982131 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 4697
I20260806 02:45:32.997779 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 4333
I20260806 02:45:32.999053 4395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:45:33.106873 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 4333
2026-08-06T02:45:33Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3769 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:45:33 dist-test-slave-8tdl krb5kdc[4843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:45:33 dist-test-slave-8tdl krb5kdc[4843](info): set up 2 sockets
Aug 06 02:45:33 dist-test-slave-8tdl krb5kdc[4843](info): commencing operation
krb5kdc: starting...
W20260806 02:45:34.238253 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.007s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:45:34 dist-test-slave-8tdl krb5kdc[4843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:45:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
I20260806 02:45:34.456312 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:43043
--webserver_interface=127.1.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:44507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:43043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:34.796600 4856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:34.797019 4856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:34.797190 4856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:34.804692 4856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:34.804817 4856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:34.804910 4856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:34.805006 4856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:34.805049 4856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:45:34.815448 4856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:44507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:43043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:43043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:34.816946 4856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:34.818387 4856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:34.830739 4861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:34.830883 4864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:34.830919 4862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:34.831979 4856 server_base.cc:1034] running on GCE node
I20260806 02:45:34.832823 4856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:34.834383 4856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:34.835616 4856 hybrid_clock.cc:648] HybridClock initialized: now 1785984334835565 us; error 38 us; skew 500 ppm
I20260806 02:45:34.836032 4856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:45:34 dist-test-slave-8tdl krb5kdc[4843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984334, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:34.839723 4856 init.cc:377] Logged in from keytab as kudu/127.1.210.126@KRBTEST.COM (short username kudu)
I20260806 02:45:34.841413 4856 webserver.cc:460] Webserver started at http://127.1.210.126:34601/ using document root <none> and password file <none>
I20260806 02:45:34.842211 4856 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:34.842382 4856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:34.842814 4856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:34.846379 4856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "b06f6e6ecda64bc1b650275f313cebd3"
format_stamp: "Formatted at 2026-08-06 02:45:34 on dist-test-slave-8tdl"
server_key: "5173a98d9b88a1cd91498fb84fec1252"
server_key_iv: "86f4da162385da57f2a29060345b91e0"
server_key_version: "encryptionkey@0"
I20260806 02:45:34.847462 4856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "b06f6e6ecda64bc1b650275f313cebd3"
format_stamp: "Formatted at 2026-08-06 02:45:34 on dist-test-slave-8tdl"
server_key: "5173a98d9b88a1cd91498fb84fec1252"
server_key_iv: "86f4da162385da57f2a29060345b91e0"
server_key_version: "encryptionkey@0"
I20260806 02:45:34.853143 4856 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260806 02:45:34.856984 4871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:34.858553 4856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:45:34.858803 4856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "b06f6e6ecda64bc1b650275f313cebd3"
format_stamp: "Formatted at 2026-08-06 02:45:34 on dist-test-slave-8tdl"
server_key: "5173a98d9b88a1cd91498fb84fec1252"
server_key_iv: "86f4da162385da57f2a29060345b91e0"
server_key_version: "encryptionkey@0"
I20260806 02:45:34.859055 4856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:34.878119 4856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:34.883232 4856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:34.883566 4856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:34.903213 4856 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:43043
I20260806 02:45:34.903244 4923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:43043 every 8 connection(s)
I20260806 02:45:34.904780 4856 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:45:34.908114 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 4856
I20260806 02:45:34.908340 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:45:34.908636 4924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:34.909157 1865 external_mini_cluster.cc:1428] Setting key 7b5983a7b1a28be7bb63a59265c63878
I20260806 02:45:34.923404 4924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b06f6e6ecda64bc1b650275f313cebd3: Bootstrap starting.
Aug 06 02:45:34 dist-test-slave-8tdl krb5kdc[4843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:34.926863 4924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b06f6e6ecda64bc1b650275f313cebd3: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:34.928399 4924 log.cc:824] T 00000000000000000000000000000000 P b06f6e6ecda64bc1b650275f313cebd3: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:34.931911 4924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b06f6e6ecda64bc1b650275f313cebd3: No bootstrap required, opened a new log
I20260806 02:45:34.935187 4927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:34.911238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57444 (local address 127.1.210.126:43043)
0806 02:45:34.912184 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:34.912200 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:34.912250 (+ 50us) server_negotiation.cc:407] Connection header received
0806 02:45:34.913915 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:34.913950 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:34.914483 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:34.915010 (+ 527us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:34.916548 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:34.918440 (+ 1892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:34.920029 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:34.921170 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:34.921375 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:34.926175 (+ 4800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:34.926186 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:34.926204 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:34.926256 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:34.930217 (+ 3961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:34.931150 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:34.931159 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:34.931168 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:34.931385 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:34.931763 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:34.931768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:34.931771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:34.932526 (+ 755us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:34.932850 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:34.933233 (+ 383us) server_negotiation.cc:299] Negotiation successful
0806 02:45:34.933693 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":407,"threads_started":1}
I20260806 02:45:34.939778 1865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:45:34.941537 4924 raft_consensus.cc:348] T 00000000000000000000000000000000 P b06f6e6ecda64bc1b650275f313cebd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b06f6e6ecda64bc1b650275f313cebd3" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 43043 } }
I20260806 02:45:34.941885 4924 raft_consensus.cc:374] T 00000000000000000000000000000000 P b06f6e6ecda64bc1b650275f313cebd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:34.942029 4924 raft_consensus.cc:729] T 00000000000000000000000000000000 P b06f6e6ecda64bc1b650275f313cebd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b06f6e6ecda64bc1b650275f313cebd3, State: Initialized, Role: FOLLOWER
I20260806 02:45:34.942785 4924 consensus_queue.cc:260] T 00000000000000000000000000000000 P b06f6e6ecda64bc1b650275f313cebd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b06f6e6ecda64bc1b650275f313cebd3" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 43043 } }
I20260806 02:45:34.943019 4924 raft_consensus.cc:388] T 00000000000000000000000000000000 P b06f6e6ecda64bc1b650275f313cebd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:45:34.943172 4924 raft_consensus.cc:482] T 00000000000000000000000000000000 P b06f6e6ecda64bc1b650275f313cebd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:45:34.943348 4924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b06f6e6ecda64bc1b650275f313cebd3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:34.945542 4924 raft_consensus.cc:504] T 00000000000000000000000000000000 P b06f6e6ecda64bc1b650275f313cebd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b06f6e6ecda64bc1b650275f313cebd3" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 43043 } }
I20260806 02:45:34.946074 4924 leader_election.cc:302] T 00000000000000000000000000000000 P b06f6e6ecda64bc1b650275f313cebd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b06f6e6ecda64bc1b650275f313cebd3; no voters:
I20260806 02:45:34.946797 4924 leader_election.cc:288] T 00000000000000000000000000000000 P b06f6e6ecda64bc1b650275f313cebd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:45:34.946907 4929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b06f6e6ecda64bc1b650275f313cebd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:34.948134 4929 raft_consensus.cc:686] T 00000000000000000000000000000000 P b06f6e6ecda64bc1b650275f313cebd3 [term 1 LEADER]: Becoming Leader. State: Replica: b06f6e6ecda64bc1b650275f313cebd3, State: Running, Role: LEADER
I20260806 02:45:34.948781 4929 consensus_queue.cc:237] T 00000000000000000000000000000000 P b06f6e6ecda64bc1b650275f313cebd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b06f6e6ecda64bc1b650275f313cebd3" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 43043 } }
I20260806 02:45:34.949358 4924 sys_catalog.cc:564] T 00000000000000000000000000000000 P b06f6e6ecda64bc1b650275f313cebd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:34.951905 4930 sys_catalog.cc:455] T 00000000000000000000000000000000 P b06f6e6ecda64bc1b650275f313cebd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b06f6e6ecda64bc1b650275f313cebd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b06f6e6ecda64bc1b650275f313cebd3" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 43043 } } }
I20260806 02:45:34.952328 4930 sys_catalog.cc:458] T 00000000000000000000000000000000 P b06f6e6ecda64bc1b650275f313cebd3 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:34.956113 4932 sys_catalog.cc:455] T 00000000000000000000000000000000 P b06f6e6ecda64bc1b650275f313cebd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b06f6e6ecda64bc1b650275f313cebd3. Latest consensus state: current_term: 1 leader_uuid: "b06f6e6ecda64bc1b650275f313cebd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b06f6e6ecda64bc1b650275f313cebd3" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 43043 } } }
I20260806 02:45:34.956626 4932 sys_catalog.cc:458] T 00000000000000000000000000000000 P b06f6e6ecda64bc1b650275f313cebd3 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:34.958549 4935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:34.966202 4935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:45:34.978633 4948 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 02:45:34.981382 4935 catalog_manager.cc:1347] Generated new cluster ID: 590e691026f54074bfb84e4b35e7e1dc
I20260806 02:45:34.981523 4935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:34.999321 4935 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:45:35.000468 4927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:34.989647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57448 (local address 127.1.210.126:43043)
0806 02:45:34.989814 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:34.989820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:34.990433 (+ 613us) server_negotiation.cc:407] Connection header received
0806 02:45:34.990605 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:34.990613 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:34.990746 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:34.990877 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:34.992076 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:34.993435 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:34.995292 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:34.996049 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:34.996201 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:34.998357 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:34.998376 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:34.998388 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:34.999461 (+ 1073us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:34.999549 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:34.999586 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:34.999727 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:34.999891 (+ 164us) server_negotiation.cc:299] Negotiation successful
0806 02:45:35.000017 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":57}
I20260806 02:45:35.001084 4935 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:35.012833 4935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b06f6e6ecda64bc1b650275f313cebd3: Generated new TSK 0
I20260806 02:45:35.014160 4935 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:45:35.024298 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 4856
I20260806 02:45:35.025974 4919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:45:35.117508 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 4856
2026-08-06T02:45:35Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2000 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:45:35 dist-test-slave-8tdl krb5kdc[4958](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:45:35 dist-test-slave-8tdl krb5kdc[4958](info): set up 2 sockets
Aug 06 02:45:35 dist-test-slave-8tdl krb5kdc[4958](info): commencing operation
krb5kdc: starting...
W20260806 02:45:36.214411 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:45:36 dist-test-slave-8tdl krb5kdc[4958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:45:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
I20260806 02:45:36.440651 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:39573
--webserver_interface=127.1.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:46491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:39573
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:36.787894 4971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:36.788398 4971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:36.788594 4971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:36.801103 4971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:36.801261 4971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:36.801345 4971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:36.801426 4971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:36.801489 4971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:45:36.821225 4971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:46491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:39573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:39573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:36.823087 4971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:36.824767 4971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:36.837531 4977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:36.837571 4979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:36.837620 4976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:36.838711 4971 server_base.cc:1034] running on GCE node
I20260806 02:45:36.839600 4971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:36.841145 4971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:36.842382 4971 hybrid_clock.cc:648] HybridClock initialized: now 1785984336842337 us; error 70 us; skew 500 ppm
I20260806 02:45:36.842818 4971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:45:36 dist-test-slave-8tdl krb5kdc[4958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984336, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:36.845950 4971 init.cc:377] Logged in from keytab as kudu/127.1.210.126@KRBTEST.COM (short username kudu)
I20260806 02:45:36.847666 4971 webserver.cc:460] Webserver started at http://127.1.210.126:42733/ using document root <none> and password file <none>
I20260806 02:45:36.848363 4971 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:36.848515 4971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:36.848831 4971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:36.852041 4971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "9d4405f539f54b85938ae51e77c6ad93"
format_stamp: "Formatted at 2026-08-06 02:45:36 on dist-test-slave-8tdl"
server_key: "4d051743895754902aa1a80ef60c3495"
server_key_iv: "5ea518d16b707f6b22a6996d32220625"
server_key_version: "encryptionkey@0"
I20260806 02:45:36.852837 4971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "9d4405f539f54b85938ae51e77c6ad93"
format_stamp: "Formatted at 2026-08-06 02:45:36 on dist-test-slave-8tdl"
server_key: "4d051743895754902aa1a80ef60c3495"
server_key_iv: "5ea518d16b707f6b22a6996d32220625"
server_key_version: "encryptionkey@0"
I20260806 02:45:36.858435 4971 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 02:45:36.862473 4986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:36.864004 4971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:36.864250 4971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "9d4405f539f54b85938ae51e77c6ad93"
format_stamp: "Formatted at 2026-08-06 02:45:36 on dist-test-slave-8tdl"
server_key: "4d051743895754902aa1a80ef60c3495"
server_key_iv: "5ea518d16b707f6b22a6996d32220625"
server_key_version: "encryptionkey@0"
I20260806 02:45:36.864470 4971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:36.883174 4971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:36.888163 4971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:36.888599 4971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:36.910575 4971 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:39573
I20260806 02:45:36.910634 5037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:39573 every 8 connection(s)
I20260806 02:45:36.912074 4971 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:45:36.913652 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 4971
I20260806 02:45:36.914021 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:45:36.915135 1865 external_mini_cluster.cc:1428] Setting key 672f3d69a37d7eba008b8224dc261ebf
I20260806 02:45:36.917578 5038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 02:45:36 dist-test-slave-8tdl krb5kdc[4958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:36.931839 5038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d4405f539f54b85938ae51e77c6ad93: Bootstrap starting.
I20260806 02:45:36.936287 5038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d4405f539f54b85938ae51e77c6ad93: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:36.937528 5038 log.cc:824] T 00000000000000000000000000000000 P 9d4405f539f54b85938ae51e77c6ad93: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:36.941231 5041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:36.917380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41530 (local address 127.1.210.126:39573)
0806 02:45:36.918226 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:36.918238 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:36.918280 (+ 42us) server_negotiation.cc:407] Connection header received
0806 02:45:36.919932 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:36.919954 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:36.920369 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:36.920714 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:36.922529 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:36.924028 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:36.925849 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:36.926758 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:36.926902 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:36.931792 (+ 4890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:36.931805 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:36.931833 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:36.931883 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:36.935951 (+ 4068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:36.936895 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:36.936903 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:36.936912 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:36.937129 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:36.937562 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:36.937567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:36.937570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:36.938346 (+ 776us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:36.938598 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:36.939399 (+ 801us) server_negotiation.cc:299] Negotiation successful
0806 02:45:36.939873 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":419,"threads_started":1}
I20260806 02:45:36.941450 5038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d4405f539f54b85938ae51e77c6ad93: No bootstrap required, opened a new log
I20260806 02:45:36.949749 5038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d4405f539f54b85938ae51e77c6ad93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d4405f539f54b85938ae51e77c6ad93" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39573 } }
I20260806 02:45:36.950122 5038 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d4405f539f54b85938ae51e77c6ad93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:36.950268 5038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d4405f539f54b85938ae51e77c6ad93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d4405f539f54b85938ae51e77c6ad93, State: Initialized, Role: FOLLOWER
I20260806 02:45:36.951007 5038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d4405f539f54b85938ae51e77c6ad93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d4405f539f54b85938ae51e77c6ad93" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39573 } }
I20260806 02:45:36.951254 5038 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d4405f539f54b85938ae51e77c6ad93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:45:36.951360 5038 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d4405f539f54b85938ae51e77c6ad93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:45:36.951503 5038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d4405f539f54b85938ae51e77c6ad93 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:36.953624 5038 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d4405f539f54b85938ae51e77c6ad93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d4405f539f54b85938ae51e77c6ad93" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39573 } }
I20260806 02:45:36.954077 5038 leader_election.cc:302] T 00000000000000000000000000000000 P 9d4405f539f54b85938ae51e77c6ad93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d4405f539f54b85938ae51e77c6ad93; no voters:
I20260806 02:45:36.954614 5038 leader_election.cc:288] T 00000000000000000000000000000000 P 9d4405f539f54b85938ae51e77c6ad93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:45:36.954793 5043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d4405f539f54b85938ae51e77c6ad93 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:36.955554 5043 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d4405f539f54b85938ae51e77c6ad93 [term 1 LEADER]: Becoming Leader. State: Replica: 9d4405f539f54b85938ae51e77c6ad93, State: Running, Role: LEADER
I20260806 02:45:36.956113 5043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d4405f539f54b85938ae51e77c6ad93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d4405f539f54b85938ae51e77c6ad93" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39573 } }
I20260806 02:45:36.956848 5038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d4405f539f54b85938ae51e77c6ad93 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:36.958894 5044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d4405f539f54b85938ae51e77c6ad93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d4405f539f54b85938ae51e77c6ad93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d4405f539f54b85938ae51e77c6ad93" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39573 } } }
I20260806 02:45:36.959280 5044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d4405f539f54b85938ae51e77c6ad93 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:36.959403 5045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d4405f539f54b85938ae51e77c6ad93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d4405f539f54b85938ae51e77c6ad93. Latest consensus state: current_term: 1 leader_uuid: "9d4405f539f54b85938ae51e77c6ad93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d4405f539f54b85938ae51e77c6ad93" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39573 } } }
I20260806 02:45:36.959905 5045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d4405f539f54b85938ae51e77c6ad93 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:36.962164 5049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:36.968197 5049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:45:36.977510 5049 catalog_manager.cc:1347] Generated new cluster ID: 41158b7914d646118ce3d08de622e85b
I20260806 02:45:36.977643 5049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:36.998070 5049 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:45:36.999802 5049 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:37.011171 5049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d4405f539f54b85938ae51e77c6ad93: Generated new TSK 0
I20260806 02:45:37.012194 5049 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab.
Entry for principal kudu/127.1.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab.
I20260806 02:45:37.083434 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.65:0
--local_ip_for_outbound_sockets=127.1.210.65
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:39573
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:46491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:37.430634 5064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:37.431042 5064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:37.431232 5064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:37.439828 5064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:37.440021 5064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:37.440213 5064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.65
I20260806 02:45:37.451582 5064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:46491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:39573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.65
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:37.453023 5064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:37.454447 5064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:37.468600 5069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:37.468647 5072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:37.469429 5064 server_base.cc:1034] running on GCE node
W20260806 02:45:37.469481 5070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:37.470196 5064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:37.471356 5064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:37.472684 5064 hybrid_clock.cc:648] HybridClock initialized: now 1785984337472633 us; error 51 us; skew 500 ppm
I20260806 02:45:37.473058 5064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:45:37 dist-test-slave-8tdl krb5kdc[4958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984337, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:37.476863 5064 init.cc:377] Logged in from keytab as kudu/127.1.210.65@KRBTEST.COM (short username kudu)
I20260806 02:45:37.478569 5064 webserver.cc:460] Webserver started at http://127.1.210.65:44043/ using document root <none> and password file <none>
I20260806 02:45:37.479364 5064 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:37.479524 5064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:37.479846 5064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:37.482996 5064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-0/data/instance:
uuid: "a0d3f08f15244c09bf05ea76345b26ef"
format_stamp: "Formatted at 2026-08-06 02:45:37 on dist-test-slave-8tdl"
server_key: "059824f4ed5c29ec9ccac1c463e0b646"
server_key_iv: "fd3299a0af1e3f289152d24ae46b4e42"
server_key_version: "encryptionkey@0"
I20260806 02:45:37.483839 5064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance:
uuid: "a0d3f08f15244c09bf05ea76345b26ef"
format_stamp: "Formatted at 2026-08-06 02:45:37 on dist-test-slave-8tdl"
server_key: "059824f4ed5c29ec9ccac1c463e0b646"
server_key_iv: "fd3299a0af1e3f289152d24ae46b4e42"
server_key_version: "encryptionkey@0"
I20260806 02:45:37.489385 5064 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 02:45:37.493326 5079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:37.494819 5064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 02:45:37.495019 5064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-0/wal
uuid: "a0d3f08f15244c09bf05ea76345b26ef"
format_stamp: "Formatted at 2026-08-06 02:45:37 on dist-test-slave-8tdl"
server_key: "059824f4ed5c29ec9ccac1c463e0b646"
server_key_iv: "fd3299a0af1e3f289152d24ae46b4e42"
server_key_version: "encryptionkey@0"
I20260806 02:45:37.495247 5064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:37.520853 5064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:37.525882 5064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:37.526247 5064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:37.527418 5064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:37.528995 5064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:37.529111 5064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:37.529269 5064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:37.529345 5064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:37.565459 5064 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.65:34965
I20260806 02:45:37.565507 5191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.65:34965 every 8 connection(s)
I20260806 02:45:37.566963 5064 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
I20260806 02:45:37.567853 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 5064
I20260806 02:45:37.568120 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance
I20260806 02:45:37.569022 1865 external_mini_cluster.cc:1428] Setting key 2fb20edec77603c6b6e0ebee49ca9c6c
Aug 06 02:45:37 dist-test-slave-8tdl krb5kdc[4958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984337, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.210.66@KRBTEST.COM; defaulting to no policy
I20260806 02:45:37.592522 5194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:37.570300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:50105 (local address 127.1.210.126:39573)
0806 02:45:37.570804 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:37.570808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:37.571963 (+ 1155us) server_negotiation.cc:407] Connection header received
0806 02:45:37.575086 (+ 3123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:37.575091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:37.575178 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:37.575284 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:37.577655 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:37.578435 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:37.580229 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:37.581036 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:37.581204 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:37.585064 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:37.585074 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:37.585082 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:37.585133 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:37.588539 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:37.589650 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:37.589655 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:37.589657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:37.589791 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:37.590400 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:37.590404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:37.590406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:37.590815 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:37.590945 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:37.592052 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0806 02:45:37.592237 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":279,"threads_started":1}
I20260806 02:45:37.593537 5195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:37.570794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:39573 (local address 127.1.210.65:50105)
0806 02:45:37.571770 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:37.571834 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:37.574697 (+ 2863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:37.575609 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:37.575624 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:37.576524 (+ 900us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:37.577428 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:37.577457 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:37.578646 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:37.578655 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:37.580041 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:37.580054 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:37.581358 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:37.581366 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:37.581655 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:37.582395 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:37.582420 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:37.584782 (+ 2362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:37.588713 (+ 3931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:37.588726 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:37.588752 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:37.589468 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:37.589918 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:37.589923 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:37.589927 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:37.590222 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:37.590968 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:37.590978 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:37.591507 (+ 529us) client_negotiation.cc:769] Sending connection context
0806 02:45:37.591848 (+ 341us) client_negotiation.cc:240] Negotiation successful
0806 02:45:37.592367 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":357,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.66@KRBTEST.COM" created.
I20260806 02:45:37.595384 5192 heartbeater.cc:347] Connected to a master server at 127.1.210.126:39573
I20260806 02:45:37.595777 5192 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:37.596881 5192 heartbeater.cc:511] Master 127.1.210.126:39573 requested a full tablet report, sending...
I20260806 02:45:37.599689 5003 ts_manager.cc:194] Registered new tserver with Master: a0d3f08f15244c09bf05ea76345b26ef (127.1.210.65:34965)
I20260806 02:45:37.601267 5003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.65@KRBTEST.COM'} at 127.1.210.65:50105
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab.
Entry for principal kudu/127.1.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab.
I20260806 02:45:37.623257 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.66:0
--local_ip_for_outbound_sockets=127.1.210.66
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:39573
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:46491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:37.967026 5198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:37.967366 5198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:37.967540 5198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:37.975214 5198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:37.975375 5198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:37.975553 5198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.66
I20260806 02:45:37.985646 5198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:46491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:39573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.66
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:37.987161 5198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:37.988637 5198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:38.003665 5203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:38.003566 5206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:38.003635 5204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:38.004288 5198 server_base.cc:1034] running on GCE node
I20260806 02:45:38.005033 5198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:38.006196 5198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:38.007507 5198 hybrid_clock.cc:648] HybridClock initialized: now 1785984338007447 us; error 54 us; skew 500 ppm
I20260806 02:45:38.008031 5198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:45:38 dist-test-slave-8tdl krb5kdc[4958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984338, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:38.011968 5198 init.cc:377] Logged in from keytab as kudu/127.1.210.66@KRBTEST.COM (short username kudu)
I20260806 02:45:38.014281 5198 webserver.cc:460] Webserver started at http://127.1.210.66:40705/ using document root <none> and password file <none>
I20260806 02:45:38.015234 5198 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:38.015396 5198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:38.015789 5198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:38.019394 5198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-1/data/instance:
uuid: "9d3a410332a741cca63bfdd01dc2ac9e"
format_stamp: "Formatted at 2026-08-06 02:45:38 on dist-test-slave-8tdl"
server_key: "cea3793f1c245216fdc2f9de2f1d60fe"
server_key_iv: "f5619f5c294149338238e122b568a7d2"
server_key_version: "encryptionkey@0"
I20260806 02:45:38.020392 5198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance:
uuid: "9d3a410332a741cca63bfdd01dc2ac9e"
format_stamp: "Formatted at 2026-08-06 02:45:38 on dist-test-slave-8tdl"
server_key: "cea3793f1c245216fdc2f9de2f1d60fe"
server_key_iv: "f5619f5c294149338238e122b568a7d2"
server_key_version: "encryptionkey@0"
I20260806 02:45:38.026726 5198 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260806 02:45:38.031210 5213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:38.032830 5198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:45:38.033043 5198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-1/wal
uuid: "9d3a410332a741cca63bfdd01dc2ac9e"
format_stamp: "Formatted at 2026-08-06 02:45:38 on dist-test-slave-8tdl"
server_key: "cea3793f1c245216fdc2f9de2f1d60fe"
server_key_iv: "f5619f5c294149338238e122b568a7d2"
server_key_version: "encryptionkey@0"
I20260806 02:45:38.033285 5198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:38.055929 5198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:38.060745 5198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:38.061086 5198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:38.062216 5198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:38.064020 5198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:38.064100 5198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:38.064236 5198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:38.064289 5198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:38.100724 5198 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.66:34877
I20260806 02:45:38.100833 5326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.66:34877 every 8 connection(s)
I20260806 02:45:38.102106 5198 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
I20260806 02:45:38.105796 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 5198
I20260806 02:45:38.106048 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance
I20260806 02:45:38.106717 1865 external_mini_cluster.cc:1428] Setting key e4895315360e783cd7e8d3f405374ad4
Aug 06 02:45:38 dist-test-slave-8tdl krb5kdc[4958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984338, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:38.124719 5329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:38.104928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:41537 (local address 127.1.210.126:39573)
0806 02:45:38.105432 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:38.105437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:38.106666 (+ 1229us) server_negotiation.cc:407] Connection header received
0806 02:45:38.108743 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:38.108748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:38.108852 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:38.108981 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:38.110626 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:38.111444 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:38.113128 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:38.113592 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:38.113736 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:38.118033 (+ 4297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:38.118040 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:38.118044 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:38.118082 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:38.121141 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:38.122034 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:38.122043 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:38.122047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:38.122206 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:38.122750 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:38.122754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:38.122756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:38.123053 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:38.123223 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:38.124005 (+ 782us) server_negotiation.cc:299] Negotiation successful
0806 02:45:38.124271 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":280,"threads_started":1}
I20260806 02:45:38.125819 5330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:38.105558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:39573 (local address 127.1.210.66:41537)
0806 02:45:38.106482 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:38.106528 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:38.108439 (+ 1911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:38.109251 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:38.109261 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:38.109817 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:38.110420 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:38.110438 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:38.111636 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:38.111642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:38.112944 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:38.112957 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:38.113865 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:38.113873 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:38.114130 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:38.115083 (+ 953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:38.115114 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:38.117737 (+ 2623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:38.121348 (+ 3611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:38.121359 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:38.121380 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:38.121887 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:38.122338 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:38.122343 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:38.122348 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:38.122572 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:38.123205 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:38.123214 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:38.123609 (+ 395us) client_negotiation.cc:769] Sending connection context
0806 02:45:38.123923 (+ 314us) client_negotiation.cc:240] Negotiation successful
0806 02:45:38.124554 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":393,"threads_started":1}
WARNING: no policy specified for kudu/127.1.210.67@KRBTEST.COM; defaulting to no policy
I20260806 02:45:38.127516 5327 heartbeater.cc:347] Connected to a master server at 127.1.210.126:39573
I20260806 02:45:38.127841 5327 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:38.128645 5327 heartbeater.cc:511] Master 127.1.210.126:39573 requested a full tablet report, sending...
I20260806 02:45:38.130280 5003 ts_manager.cc:194] Registered new tserver with Master: 9d3a410332a741cca63bfdd01dc2ac9e (127.1.210.66:34877)
I20260806 02:45:38.131240 5003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.66@KRBTEST.COM'} at 127.1.210.66:41537
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab.
Entry for principal kudu/127.1.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab.
I20260806 02:45:38.171005 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.67:0
--local_ip_for_outbound_sockets=127.1.210.67
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:39573
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:46491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:38.521169 5333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:38.521505 5333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:38.521674 5333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:38.529172 5333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:38.529348 5333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:38.529542 5333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.67
I20260806 02:45:38.539809 5333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:46491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:39573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.67
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:38.541290 5333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:38.542723 5333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:38.556583 5338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:38.557266 5341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:38.557457 5339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:38.557893 5333 server_base.cc:1034] running on GCE node
I20260806 02:45:38.558626 5333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:38.559700 5333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:38.560974 5333 hybrid_clock.cc:648] HybridClock initialized: now 1785984338560909 us; error 46 us; skew 500 ppm
I20260806 02:45:38.561395 5333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:45:38 dist-test-slave-8tdl krb5kdc[4958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984338, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:38.564877 5333 init.cc:377] Logged in from keytab as kudu/127.1.210.67@KRBTEST.COM (short username kudu)
I20260806 02:45:38.566645 5333 webserver.cc:460] Webserver started at http://127.1.210.67:40439/ using document root <none> and password file <none>
I20260806 02:45:38.567420 5333 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:38.567551 5333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:38.567874 5333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:38.570995 5333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-2/data/instance:
uuid: "111517becbcf4bfe847e8ad352b86537"
format_stamp: "Formatted at 2026-08-06 02:45:38 on dist-test-slave-8tdl"
server_key: "5259c9db46b8e3bd135f71f496dee378"
server_key_iv: "b7297f124730ecd3e5851a9ac31e69d0"
server_key_version: "encryptionkey@0"
I20260806 02:45:38.571866 5333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance:
uuid: "111517becbcf4bfe847e8ad352b86537"
format_stamp: "Formatted at 2026-08-06 02:45:38 on dist-test-slave-8tdl"
server_key: "5259c9db46b8e3bd135f71f496dee378"
server_key_iv: "b7297f124730ecd3e5851a9ac31e69d0"
server_key_version: "encryptionkey@0"
I20260806 02:45:38.577296 5333 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260806 02:45:38.581019 5348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:38.582511 5333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:45:38.582726 5333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-2/wal
uuid: "111517becbcf4bfe847e8ad352b86537"
format_stamp: "Formatted at 2026-08-06 02:45:38 on dist-test-slave-8tdl"
server_key: "5259c9db46b8e3bd135f71f496dee378"
server_key_iv: "b7297f124730ecd3e5851a9ac31e69d0"
server_key_version: "encryptionkey@0"
I20260806 02:45:38.582981 5333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:38.604496 5192 heartbeater.cc:503] Master 127.1.210.126:39573 was elected leader, sending a full tablet report...
I20260806 02:45:38.604920 5333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:38.610189 5333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:38.610560 5333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:38.611689 5333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:38.613372 5333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:38.613454 5333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:38.613550 5333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:38.613600 5333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:38.651446 5333 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.67:41045
I20260806 02:45:38.651481 5460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.67:41045 every 8 connection(s)
I20260806 02:45:38.653045 5333 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
I20260806 02:45:38.663278 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 5333
I20260806 02:45:38.663486 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance
I20260806 02:45:38.664129 1865 external_mini_cluster.cc:1428] Setting key 7873e3f16c92c99739755bdebcf4c952
Aug 06 02:45:38 dist-test-slave-8tdl krb5kdc[4958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984338, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:38.676882 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:38.656002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:58313 (local address 127.1.210.126:39573)
0806 02:45:38.656544 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:38.656548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:38.657496 (+ 948us) server_negotiation.cc:407] Connection header received
0806 02:45:38.659765 (+ 2269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:38.659771 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:38.659897 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:38.660029 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:38.662033 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:38.663188 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:38.665494 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:38.666096 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:38.666227 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:38.670830 (+ 4603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:38.670835 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:38.670839 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:38.670875 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:38.673539 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:38.674394 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:38.674398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:38.674400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:38.674538 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:38.675096 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:38.675101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:38.675103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:38.675440 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:38.675578 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:38.676377 (+ 799us) server_negotiation.cc:299] Negotiation successful
0806 02:45:38.676591 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":281,"threads_started":1}
I20260806 02:45:38.677987 5464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:38.656498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:39573 (local address 127.1.210.67:58313)
0806 02:45:38.657319 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:38.657367 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:38.659433 (+ 2066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:38.660313 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:38.660323 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:38.660952 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:38.661721 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:38.661746 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:38.663443 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:38.663505 (+ 62us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:38.665330 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:38.665346 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:38.666398 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:38.666410 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:38.666815 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:38.667832 (+ 1017us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:38.667888 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:38.670531 (+ 2643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:38.673692 (+ 3161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:38.673701 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:38.673717 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:38.674263 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:38.674659 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:38.674662 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:38.674665 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:38.674939 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:38.675563 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:38.675570 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:38.675959 (+ 389us) client_negotiation.cc:769] Sending connection context
0806 02:45:38.676225 (+ 266us) client_negotiation.cc:240] Negotiation successful
0806 02:45:38.676727 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":289,"threads_started":1}
I20260806 02:45:38.679787 5461 heartbeater.cc:347] Connected to a master server at 127.1.210.126:39573
I20260806 02:45:38.680106 5461 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:38.680900 5461 heartbeater.cc:511] Master 127.1.210.126:39573 requested a full tablet report, sending...
I20260806 02:45:38.682408 5003 ts_manager.cc:194] Registered new tserver with Master: 111517becbcf4bfe847e8ad352b86537 (127.1.210.67:41045)
I20260806 02:45:38.683317 5003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.67@KRBTEST.COM'} at 127.1.210.67:58313
I20260806 02:45:38.695168 1865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:45:38 dist-test-slave-8tdl krb5kdc[4958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984338, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 06 02:45:38 dist-test-slave-8tdl krb5kdc[4958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984338, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:38.751251 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:38.735044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41546 (local address 127.1.210.126:39573)
0806 02:45:38.735367 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:38.735377 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:38.736143 (+ 766us) server_negotiation.cc:407] Connection header received
0806 02:45:38.736372 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:38.736377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:38.736486 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:38.736619 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:38.738569 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:38.739413 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:38.740695 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:38.741117 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:38.741265 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:38.745503 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:38.745509 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:38.745512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:38.745552 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:38.748314 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:38.749161 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:38.749167 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:38.749173 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:38.749316 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:38.749724 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:38.749729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:38.749732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:38.750082 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:38.750351 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:38.750654 (+ 303us) server_negotiation.cc:299] Negotiation successful
0806 02:45:38.750920 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260806 02:45:38.751838 5003 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41546
I20260806 02:45:38.753928 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 5064
Aug 06 02:45:38 dist-test-slave-8tdl krb5kdc[4958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984338, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.210.65@KRBTEST.COM
I20260806 02:45:38.772414 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:38.755169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34292 (local address 127.1.210.65:34965)
0806 02:45:38.755809 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:38.755818 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:38.755845 (+ 27us) server_negotiation.cc:407] Connection header received
0806 02:45:38.755906 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:38.755913 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:38.756153 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:38.756289 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:38.757597 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:38.758870 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:38.760131 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:38.760762 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:38.760872 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:38.765580 (+ 4708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:38.765598 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:38.765617 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:38.765666 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:38.769148 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:38.770154 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:38.770163 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:38.770166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:38.770313 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:38.770764 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:38.770767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:38.770770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:38.771183 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:38.771426 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:38.771721 (+ 295us) server_negotiation.cc:299] Negotiation successful
0806 02:45:38.771944 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":323,"threads_started":1}
W20260806 02:45:38.773916 5187 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34292
W20260806 02:45:38.775280 1865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 5064: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:45:38.775507 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 5064
I20260806 02:45:38.793542 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 5198
Aug 06 02:45:38 dist-test-slave-8tdl krb5kdc[4958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984338, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.210.66@KRBTEST.COM
I20260806 02:45:38.810647 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:38.794453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44212 (local address 127.1.210.66:34877)
0806 02:45:38.795185 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:38.795192 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:38.795220 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:45:38.795288 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:38.795295 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:38.795509 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:38.795665 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:38.796905 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:38.798077 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:38.799539 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:38.800152 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:38.800266 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:38.804562 (+ 4296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:38.804575 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:38.804591 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:38.804638 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:38.807729 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:38.808501 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:38.808511 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:38.808514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:38.808647 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:38.809057 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:38.809062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:38.809064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:38.809455 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:38.809710 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:38.810079 (+ 369us) server_negotiation.cc:299] Negotiation successful
0806 02:45:38.810344 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":415,"threads_started":1}
W20260806 02:45:38.812119 5322 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44212
W20260806 02:45:38.813311 1865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 5198: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:45:38.813493 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 5198
I20260806 02:45:38.831640 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 5333
Aug 06 02:45:38 dist-test-slave-8tdl krb5kdc[4958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984338, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.210.67@KRBTEST.COM
I20260806 02:45:38.849676 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:38.832555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60976 (local address 127.1.210.67:41045)
0806 02:45:38.833166 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:38.833174 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:38.833203 (+ 29us) server_negotiation.cc:407] Connection header received
0806 02:45:38.833262 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:38.833268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:38.833518 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:38.833701 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:38.835242 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:38.836312 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:38.837615 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:38.838240 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:38.838390 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:38.843295 (+ 4905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:38.843310 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:38.843326 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:38.843365 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:38.846507 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:38.847419 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:38.847429 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:38.847431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:38.847576 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:38.848004 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:38.848008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:38.848010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:38.848449 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:38.848701 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:38.849000 (+ 299us) server_negotiation.cc:299] Negotiation successful
0806 02:45:38.849254 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":294,"threads_started":1}
W20260806 02:45:38.851141 5456 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60976
W20260806 02:45:38.852283 1865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 5333: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:45:38.852456 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 5333
I20260806 02:45:38.869145 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 4971
I20260806 02:45:38.870471 5033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:45:38.977017 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 4971
2026-08-06T02:45:38Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3871 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:45:39 dist-test-slave-8tdl krb5kdc[5479](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:45:39 dist-test-slave-8tdl krb5kdc[5479](info): set up 2 sockets
Aug 06 02:45:39 dist-test-slave-8tdl krb5kdc[5479](info): commencing operation
krb5kdc: starting...
W20260806 02:45:40.138553 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:45:40 dist-test-slave-8tdl krb5kdc[5479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:45:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
I20260806 02:45:40.379887 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:34727
--webserver_interface=127.1.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:44785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:34727
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:40.731166 5492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:40.731483 5492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:40.731663 5492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:40.739707 5492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:40.739846 5492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:40.739948 5492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:40.740084 5492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:40.740154 5492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:45:40.752835 5492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:44785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:34727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:34727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:40.754411 5492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:40.756063 5492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:40.770318 5497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:40.770512 5498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:40.770478 5500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:40.771719 5492 server_base.cc:1034] running on GCE node
I20260806 02:45:40.772634 5492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:40.774297 5492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:40.775532 5492 hybrid_clock.cc:648] HybridClock initialized: now 1785984340775457 us; error 44 us; skew 500 ppm
I20260806 02:45:40.775928 5492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:45:40 dist-test-slave-8tdl krb5kdc[5479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984340, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:40.779539 5492 init.cc:377] Logged in from keytab as kudu/127.1.210.126@KRBTEST.COM (short username kudu)
I20260806 02:45:40.781332 5492 webserver.cc:460] Webserver started at http://127.1.210.126:36229/ using document root <none> and password file <none>
I20260806 02:45:40.782161 5492 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:40.782317 5492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:40.782651 5492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:40.785903 5492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "17c7743b12b74bb38d9beb1bb581dbc2"
format_stamp: "Formatted at 2026-08-06 02:45:40 on dist-test-slave-8tdl"
server_key: "ee3335149c89596c77733791d6b17d22"
server_key_iv: "5ad783769dc5dbce94dd0ff44f1608f4"
server_key_version: "encryptionkey@0"
I20260806 02:45:40.786906 5492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "17c7743b12b74bb38d9beb1bb581dbc2"
format_stamp: "Formatted at 2026-08-06 02:45:40 on dist-test-slave-8tdl"
server_key: "ee3335149c89596c77733791d6b17d22"
server_key_iv: "5ad783769dc5dbce94dd0ff44f1608f4"
server_key_version: "encryptionkey@0"
I20260806 02:45:40.796025 5492 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.007s sys 0.001s
I20260806 02:45:40.800978 5507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:40.803025 5492 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 02:45:40.803298 5492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "17c7743b12b74bb38d9beb1bb581dbc2"
format_stamp: "Formatted at 2026-08-06 02:45:40 on dist-test-slave-8tdl"
server_key: "ee3335149c89596c77733791d6b17d22"
server_key_iv: "5ad783769dc5dbce94dd0ff44f1608f4"
server_key_version: "encryptionkey@0"
I20260806 02:45:40.803570 5492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:40.830396 5492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:40.835830 5492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:40.836179 5492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:40.857535 5492 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:34727
I20260806 02:45:40.857582 5558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:34727 every 8 connection(s)
I20260806 02:45:40.859094 5492 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:45:40.863271 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:40.864084 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 5492
I20260806 02:45:40.864363 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:45:40.865253 1865 external_mini_cluster.cc:1428] Setting key c4191f3eb6a373465d591dbbfc9b5708
I20260806 02:45:40.878964 5559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17c7743b12b74bb38d9beb1bb581dbc2: Bootstrap starting.
Aug 06 02:45:40 dist-test-slave-8tdl krb5kdc[5479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:40.882965 5559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17c7743b12b74bb38d9beb1bb581dbc2: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:40.884403 5559 log.cc:824] T 00000000000000000000000000000000 P 17c7743b12b74bb38d9beb1bb581dbc2: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:40.887694 5559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17c7743b12b74bb38d9beb1bb581dbc2: No bootstrap required, opened a new log
I20260806 02:45:40.891062 5562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:40.867699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58120 (local address 127.1.210.126:34727)
0806 02:45:40.868605 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:40.868619 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:40.868676 (+ 57us) server_negotiation.cc:407] Connection header received
0806 02:45:40.870412 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:40.870442 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:40.870977 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:40.871351 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:40.873257 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:40.874636 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:40.876581 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:40.877515 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:40.877679 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:40.882660 (+ 4981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:40.882669 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:40.882740 (+ 71us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:40.882794 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:40.886234 (+ 3440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:40.887163 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:40.887171 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:40.887178 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:40.887364 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:40.887807 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:40.887832 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:40.887836 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:40.888440 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:40.888800 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:40.889270 (+ 470us) server_negotiation.cc:299] Negotiation successful
0806 02:45:40.889789 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":432,"threads_started":1}
I20260806 02:45:40.896942 5559 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17c7743b12b74bb38d9beb1bb581dbc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c7743b12b74bb38d9beb1bb581dbc2" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 34727 } }
I20260806 02:45:40.897435 5559 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17c7743b12b74bb38d9beb1bb581dbc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:40.897560 5559 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17c7743b12b74bb38d9beb1bb581dbc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17c7743b12b74bb38d9beb1bb581dbc2, State: Initialized, Role: FOLLOWER
I20260806 02:45:40.898327 5559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17c7743b12b74bb38d9beb1bb581dbc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c7743b12b74bb38d9beb1bb581dbc2" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 34727 } }
I20260806 02:45:40.898636 5559 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17c7743b12b74bb38d9beb1bb581dbc2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:45:40.898819 5559 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17c7743b12b74bb38d9beb1bb581dbc2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:45:40.899066 5559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17c7743b12b74bb38d9beb1bb581dbc2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:40.901432 5559 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17c7743b12b74bb38d9beb1bb581dbc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c7743b12b74bb38d9beb1bb581dbc2" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 34727 } }
I20260806 02:45:40.901965 5559 leader_election.cc:302] T 00000000000000000000000000000000 P 17c7743b12b74bb38d9beb1bb581dbc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17c7743b12b74bb38d9beb1bb581dbc2; no voters:
I20260806 02:45:40.902632 5559 leader_election.cc:288] T 00000000000000000000000000000000 P 17c7743b12b74bb38d9beb1bb581dbc2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:45:40.902794 5564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17c7743b12b74bb38d9beb1bb581dbc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:40.904274 5564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17c7743b12b74bb38d9beb1bb581dbc2 [term 1 LEADER]: Becoming Leader. State: Replica: 17c7743b12b74bb38d9beb1bb581dbc2, State: Running, Role: LEADER
I20260806 02:45:40.904803 5564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17c7743b12b74bb38d9beb1bb581dbc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c7743b12b74bb38d9beb1bb581dbc2" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 34727 } }
I20260806 02:45:40.905150 5559 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17c7743b12b74bb38d9beb1bb581dbc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:40.908037 5566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17c7743b12b74bb38d9beb1bb581dbc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17c7743b12b74bb38d9beb1bb581dbc2. Latest consensus state: current_term: 1 leader_uuid: "17c7743b12b74bb38d9beb1bb581dbc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c7743b12b74bb38d9beb1bb581dbc2" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 34727 } } }
I20260806 02:45:40.907755 5565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17c7743b12b74bb38d9beb1bb581dbc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17c7743b12b74bb38d9beb1bb581dbc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c7743b12b74bb38d9beb1bb581dbc2" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 34727 } } }
I20260806 02:45:40.909418 5565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17c7743b12b74bb38d9beb1bb581dbc2 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:40.909418 5566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17c7743b12b74bb38d9beb1bb581dbc2 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:40.913053 5571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:40.919651 5571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:45:40.932111 5571 catalog_manager.cc:1347] Generated new cluster ID: abf56ddced3d4e4e9cc8a845e6a90985
I20260806 02:45:40.932358 5571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:40.943358 5571 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:45:40.944954 5571 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:40.953478 5571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17c7743b12b74bb38d9beb1bb581dbc2: Generated new TSK 0
I20260806 02:45:40.954432 5571 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab.
Entry for principal kudu/127.1.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab.
I20260806 02:45:41.016445 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.65:0
--local_ip_for_outbound_sockets=127.1.210.65
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:34727
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:44785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:41.380326 5585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:41.380653 5585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:41.380755 5585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:41.388505 5585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:41.388636 5585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:41.388754 5585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.65
I20260806 02:45:41.399113 5585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:44785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:34727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.65
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:41.400691 5585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:41.402050 5585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:41.419289 5590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:41.419392 5593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:41.419472 5591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:41.421010 5585 server_base.cc:1034] running on GCE node
I20260806 02:45:41.421753 5585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:41.422966 5585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:41.424232 5585 hybrid_clock.cc:648] HybridClock initialized: now 1785984341424184 us; error 37 us; skew 500 ppm
I20260806 02:45:41.424630 5585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:45:41 dist-test-slave-8tdl krb5kdc[5479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984341, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:41.428738 5585 init.cc:377] Logged in from keytab as kudu/127.1.210.65@KRBTEST.COM (short username kudu)
I20260806 02:45:41.430428 5585 webserver.cc:460] Webserver started at http://127.1.210.65:45561/ using document root <none> and password file <none>
I20260806 02:45:41.431313 5585 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:41.431440 5585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:41.431743 5585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:41.435125 5585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-0/data/instance:
uuid: "a8f57fe5068348549d0c636f672d45a3"
format_stamp: "Formatted at 2026-08-06 02:45:41 on dist-test-slave-8tdl"
server_key: "30b5f5dbc38ef85592850c52b3fc2d7f"
server_key_iv: "241d4304eb69c4a7b840ebc23aa5268c"
server_key_version: "encryptionkey@0"
I20260806 02:45:41.436031 5585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance:
uuid: "a8f57fe5068348549d0c636f672d45a3"
format_stamp: "Formatted at 2026-08-06 02:45:41 on dist-test-slave-8tdl"
server_key: "30b5f5dbc38ef85592850c52b3fc2d7f"
server_key_iv: "241d4304eb69c4a7b840ebc23aa5268c"
server_key_version: "encryptionkey@0"
I20260806 02:45:41.442111 5585 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260806 02:45:41.446450 5600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:41.448261 5585 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:45:41.448427 5585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-0/wal
uuid: "a8f57fe5068348549d0c636f672d45a3"
format_stamp: "Formatted at 2026-08-06 02:45:41 on dist-test-slave-8tdl"
server_key: "30b5f5dbc38ef85592850c52b3fc2d7f"
server_key_iv: "241d4304eb69c4a7b840ebc23aa5268c"
server_key_version: "encryptionkey@0"
I20260806 02:45:41.448627 5585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:41.471652 5585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:41.476493 5585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:41.476784 5585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:41.477939 5585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:41.479552 5585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:41.479692 5585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:41.479827 5585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:41.479880 5585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:41.516443 5585 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.65:37959
I20260806 02:45:41.516572 5712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.65:37959 every 8 connection(s)
I20260806 02:45:41.518083 5585 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
I20260806 02:45:41.520924 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 5585
I20260806 02:45:41.521164 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance
I20260806 02:45:41.522218 1865 external_mini_cluster.cc:1428] Setting key 1a9fdff1e9a4d27fb8af267899d60755
Aug 06 02:45:41 dist-test-slave-8tdl krb5kdc[5479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984341, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:41.541141 5715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:41.521073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:40729 (local address 127.1.210.126:34727)
0806 02:45:41.523436 (+ 2363us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:41.523443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:41.523466 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:45:41.525202 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:41.525208 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:41.525307 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:41.525439 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:41.527182 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:41.527923 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:41.529584 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:41.530027 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:41.530187 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:41.534627 (+ 4440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:41.534633 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:41.534636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:41.534715 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:41.537716 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:41.538559 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:41.538563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:41.538565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:41.538761 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:41.539227 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:41.539232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:41.539235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:41.539513 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:41.539686 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:41.540497 (+ 811us) server_negotiation.cc:299] Negotiation successful
0806 02:45:41.540754 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2230,"thread_start_us":2094,"threads_started":1}
I20260806 02:45:41.542207 5716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:41.521658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:34727 (local address 127.1.210.65:40729)
0806 02:45:41.522552 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:41.522619 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:41.524908 (+ 2289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:41.525725 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:41.525734 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:41.526294 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:41.526996 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:41.527015 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:41.528104 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:41.528109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:41.529421 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:41.529435 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:41.530291 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:41.530300 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:41.530554 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:41.531484 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:41.531521 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:41.534361 (+ 2840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:41.537886 (+ 3525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:41.537898 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:41.537920 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:41.538403 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:41.538880 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:41.538886 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:41.538889 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:41.539099 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:41.539669 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:41.539678 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:41.540067 (+ 389us) client_negotiation.cc:769] Sending connection context
0806 02:45:41.540403 (+ 336us) client_negotiation.cc:240] Negotiation successful
0806 02:45:41.541021 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":363,"threads_started":1}
WARNING: no policy specified for kudu/127.1.210.66@KRBTEST.COM; defaulting to no policy
I20260806 02:45:41.544080 5713 heartbeater.cc:347] Connected to a master server at 127.1.210.126:34727
I20260806 02:45:41.544376 5713 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:41.545260 5713 heartbeater.cc:511] Master 127.1.210.126:34727 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.66@KRBTEST.COM" created.
I20260806 02:45:41.547887 5524 ts_manager.cc:194] Registered new tserver with Master: a8f57fe5068348549d0c636f672d45a3 (127.1.210.65:37959)
I20260806 02:45:41.549448 5524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.65@KRBTEST.COM'} at 127.1.210.65:40729
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab.
Entry for principal kudu/127.1.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab.
I20260806 02:45:41.575206 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.66:0
--local_ip_for_outbound_sockets=127.1.210.66
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:34727
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:44785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:41.936630 5719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:41.936954 5719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:41.937077 5719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:41.944850 5719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:41.944995 5719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:41.945178 5719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.66
I20260806 02:45:41.955797 5719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:44785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:34727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.66
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:41.957278 5719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:41.958801 5719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:41.973744 5724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:41.973673 5725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:41.973702 5727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:41.974529 5719 server_base.cc:1034] running on GCE node
I20260806 02:45:41.975322 5719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:41.976368 5719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:41.977620 5719 hybrid_clock.cc:648] HybridClock initialized: now 1785984341977563 us; error 51 us; skew 500 ppm
I20260806 02:45:41.978085 5719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:45:41 dist-test-slave-8tdl krb5kdc[5479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984341, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:41.987932 5719 init.cc:377] Logged in from keytab as kudu/127.1.210.66@KRBTEST.COM (short username kudu)
I20260806 02:45:41.989868 5719 webserver.cc:460] Webserver started at http://127.1.210.66:42055/ using document root <none> and password file <none>
I20260806 02:45:41.990753 5719 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:41.990917 5719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:41.991279 5719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:41.995076 5719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-1/data/instance:
uuid: "4e3306704aa3498aa6fd6a1c36060791"
format_stamp: "Formatted at 2026-08-06 02:45:41 on dist-test-slave-8tdl"
server_key: "19e257d9c2af2dd61f39fb538ed952f1"
server_key_iv: "588209d04852cd0a79b3926de34783cb"
server_key_version: "encryptionkey@0"
I20260806 02:45:41.996095 5719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance:
uuid: "4e3306704aa3498aa6fd6a1c36060791"
format_stamp: "Formatted at 2026-08-06 02:45:41 on dist-test-slave-8tdl"
server_key: "19e257d9c2af2dd61f39fb538ed952f1"
server_key_iv: "588209d04852cd0a79b3926de34783cb"
server_key_version: "encryptionkey@0"
I20260806 02:45:42.002270 5719 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 02:45:42.006974 5734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:42.008873 5719 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:45:42.009090 5719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-1/wal
uuid: "4e3306704aa3498aa6fd6a1c36060791"
format_stamp: "Formatted at 2026-08-06 02:45:41 on dist-test-slave-8tdl"
server_key: "19e257d9c2af2dd61f39fb538ed952f1"
server_key_iv: "588209d04852cd0a79b3926de34783cb"
server_key_version: "encryptionkey@0"
I20260806 02:45:42.009334 5719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:42.030860 5719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:42.036275 5719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:42.036633 5719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:42.037781 5719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:42.039511 5719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:42.039588 5719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:42.039680 5719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:42.039726 5719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:42.078028 5719 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.66:32909
I20260806 02:45:42.078140 5846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.66:32909 every 8 connection(s)
I20260806 02:45:42.079492 5719 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
I20260806 02:45:42.089540 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 5719
I20260806 02:45:42.089771 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance
I20260806 02:45:42.090241 1865 external_mini_cluster.cc:1428] Setting key 33c87df3e88507fc3513d179a4f378db
Aug 06 02:45:42 dist-test-slave-8tdl krb5kdc[5479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984341, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:42.107322 5849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:42.082543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:34081 (local address 127.1.210.126:34727)
0806 02:45:42.083059 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:42.083064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:42.084232 (+ 1168us) server_negotiation.cc:407] Connection header received
0806 02:45:42.086288 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:42.086294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:42.086408 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:42.086553 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:42.088402 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:42.089271 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:42.092083 (+ 2812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:42.093087 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:42.093295 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:42.098306 (+ 5011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:42.098318 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:42.098326 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:42.098384 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:42.102819 (+ 4435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:42.104087 (+ 1268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:42.104094 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:42.104097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:42.104261 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:42.104893 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:42.104897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:42.104900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:42.105267 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:42.105401 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:42.106580 (+ 1179us) server_negotiation.cc:299] Negotiation successful
0806 02:45:42.107007 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":404,"thread_start_us":300,"threads_started":1}
I20260806 02:45:42.109454 5850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:42.083136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:34727 (local address 127.1.210.66:34081)
0806 02:45:42.084017 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:42.084064 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:42.085908 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:42.086885 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:42.086896 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:42.087528 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:42.088183 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:42.088202 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:42.089491 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:42.089496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:42.091792 (+ 2296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:42.091809 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:42.093403 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:42.093410 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:42.093663 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:42.094922 (+ 1259us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:42.094956 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:42.097936 (+ 2980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:42.103045 (+ 5109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:42.103060 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:42.103086 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:42.103870 (+ 784us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:42.104403 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:42.104410 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:42.104416 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:42.104722 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:42.105441 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:42.105453 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:42.106004 (+ 551us) client_negotiation.cc:769] Sending connection context
0806 02:45:42.106361 (+ 357us) client_negotiation.cc:240] Negotiation successful
0806 02:45:42.107103 (+ 742us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":329,"threads_started":1}
I20260806 02:45:42.111589 5847 heartbeater.cc:347] Connected to a master server at 127.1.210.126:34727
I20260806 02:45:42.112136 5847 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:42.113230 5847 heartbeater.cc:511] Master 127.1.210.126:34727 requested a full tablet report, sending...
I20260806 02:45:42.115449 5524 ts_manager.cc:194] Registered new tserver with Master: 4e3306704aa3498aa6fd6a1c36060791 (127.1.210.66:32909)
I20260806 02:45:42.116360 5524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.66@KRBTEST.COM'} at 127.1.210.66:34081
WARNING: no policy specified for kudu/127.1.210.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab.
Entry for principal kudu/127.1.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab.
I20260806 02:45:42.155820 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.67:0
--local_ip_for_outbound_sockets=127.1.210.67
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:34727
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:44785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:42.507229 5853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:42.507664 5853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:42.507853 5853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:42.516870 5853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:42.517050 5853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:42.517292 5853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.67
I20260806 02:45:42.528996 5853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:44785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:34727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.67
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:42.530522 5853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:42.532081 5853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:42.547334 5858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:42.547415 5861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:42.547492 5859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:42.548902 5853 server_base.cc:1034] running on GCE node
I20260806 02:45:42.549647 5853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:42.550774 5853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:42.552068 5853 hybrid_clock.cc:648] HybridClock initialized: now 1785984342551999 us; error 78 us; skew 500 ppm
I20260806 02:45:42.552448 5853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:45:42.552816 5713 heartbeater.cc:503] Master 127.1.210.126:34727 was elected leader, sending a full tablet report...
Aug 06 02:45:42 dist-test-slave-8tdl krb5kdc[5479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984342, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:42.556710 5853 init.cc:377] Logged in from keytab as kudu/127.1.210.67@KRBTEST.COM (short username kudu)
I20260806 02:45:42.558501 5853 webserver.cc:460] Webserver started at http://127.1.210.67:46485/ using document root <none> and password file <none>
I20260806 02:45:42.559329 5853 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:42.559485 5853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:42.559826 5853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:42.563060 5853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-2/data/instance:
uuid: "e89caf1d0e6c431097346949cdd38da1"
format_stamp: "Formatted at 2026-08-06 02:45:42 on dist-test-slave-8tdl"
server_key: "174edc7569628ece46c474a09f389ab3"
server_key_iv: "4b6c3cabe4016425ce6cd9da7d08f40f"
server_key_version: "encryptionkey@0"
I20260806 02:45:42.563938 5853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance:
uuid: "e89caf1d0e6c431097346949cdd38da1"
format_stamp: "Formatted at 2026-08-06 02:45:42 on dist-test-slave-8tdl"
server_key: "174edc7569628ece46c474a09f389ab3"
server_key_iv: "4b6c3cabe4016425ce6cd9da7d08f40f"
server_key_version: "encryptionkey@0"
I20260806 02:45:42.570374 5853 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260806 02:45:42.574467 5868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:42.576145 5853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:42.576355 5853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-2/wal
uuid: "e89caf1d0e6c431097346949cdd38da1"
format_stamp: "Formatted at 2026-08-06 02:45:42 on dist-test-slave-8tdl"
server_key: "174edc7569628ece46c474a09f389ab3"
server_key_iv: "4b6c3cabe4016425ce6cd9da7d08f40f"
server_key_version: "encryptionkey@0"
I20260806 02:45:42.576597 5853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:42.592041 5853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:42.596994 5853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:42.597307 5853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:42.598412 5853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:42.599908 5853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:42.600020 5853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:42.600175 5853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:42.600252 5853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:42.637765 5853 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.67:33205
I20260806 02:45:42.637864 5981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.67:33205 every 8 connection(s)
I20260806 02:45:42.639235 5853 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
I20260806 02:45:42.649649 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 5853
I20260806 02:45:42.649883 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance
I20260806 02:45:42.650432 1865 external_mini_cluster.cc:1428] Setting key 3d64f65f4348a4e46cee5e8ab512b099
Aug 06 02:45:42 dist-test-slave-8tdl krb5kdc[5479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984342, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:42.662222 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:42.642190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:55493 (local address 127.1.210.126:34727)
0806 02:45:42.642720 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:42.642724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:42.643662 (+ 938us) server_negotiation.cc:407] Connection header received
0806 02:45:42.645580 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:42.645595 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:42.645686 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:42.645812 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:42.647491 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:42.648328 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:42.650571 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:42.651149 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:42.651322 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:42.655935 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:42.655940 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:42.655943 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:42.655980 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:42.658877 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:42.659757 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:42.659761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:42.659764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:42.659900 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:42.660385 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:42.660388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:42.660390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:42.660676 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:42.660802 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:42.661727 (+ 925us) server_negotiation.cc:299] Negotiation successful
0806 02:45:42.661918 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":264,"threads_started":1}
I20260806 02:45:42.663501 5985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:42.642654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:34727 (local address 127.1.210.67:55493)
0806 02:45:42.643455 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:42.643501 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:42.645283 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:42.646069 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:42.646077 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:42.646636 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:42.647312 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:42.647330 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:42.648513 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:42.648517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:42.650338 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:42.650356 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:42.651443 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:42.651450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:42.651735 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:42.652879 (+ 1144us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:42.652915 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:42.655692 (+ 2777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:42.659048 (+ 3356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:42.659056 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:42.659071 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:42.659603 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:42.660034 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:42.660038 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:42.660040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:42.660261 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:42.660839 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:42.660847 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:42.661326 (+ 479us) client_negotiation.cc:769] Sending connection context
0806 02:45:42.661616 (+ 290us) client_negotiation.cc:240] Negotiation successful
0806 02:45:42.662106 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":307,"threads_started":1}
I20260806 02:45:42.665271 5982 heartbeater.cc:347] Connected to a master server at 127.1.210.126:34727
I20260806 02:45:42.665714 5982 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:42.666728 5982 heartbeater.cc:511] Master 127.1.210.126:34727 requested a full tablet report, sending...
I20260806 02:45:42.668372 5524 ts_manager.cc:194] Registered new tserver with Master: e89caf1d0e6c431097346949cdd38da1 (127.1.210.67:33205)
I20260806 02:45:42.669286 5524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.67@KRBTEST.COM'} at 127.1.210.67:55493
I20260806 02:45:42.680285 1865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:45:42 dist-test-slave-8tdl krb5kdc[5479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 02:45:42 dist-test-slave-8tdl krb5kdc[5479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.65@KRBTEST.COM
I20260806 02:45:42.738353 5990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:42.720440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47020 (local address 127.1.210.65:37959)
0806 02:45:42.721111 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:42.721119 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:42.721148 (+ 29us) server_negotiation.cc:407] Connection header received
0806 02:45:42.721356 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:42.721364 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:42.721601 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:42.721758 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:42.723616 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:42.724740 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:42.726194 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:42.726909 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:42.727024 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:42.731612 (+ 4588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:42.731629 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:42.731645 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:42.731691 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:42.735205 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:42.736005 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:42.736013 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:42.736016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:42.736165 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:42.736609 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:42.736614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:42.736616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:42.737063 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:42.737312 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:42.737648 (+ 336us) server_negotiation.cc:299] Negotiation successful
0806 02:45:42.737877 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":311,"threads_started":1}
Aug 06 02:45:42 dist-test-slave-8tdl krb5kdc[5479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:42.760798 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:42.744382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58136 (local address 127.1.210.126:34727)
0806 02:45:42.744624 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:42.744630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:42.744825 (+ 195us) server_negotiation.cc:407] Connection header received
0806 02:45:42.745046 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:42.745052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:42.745136 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:42.745248 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:42.746758 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:42.747528 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:42.748812 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:42.749248 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:42.749397 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:42.754816 (+ 5419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:42.754822 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:42.754825 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:42.754883 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:42.757439 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:42.758284 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:42.758288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:42.758291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:42.758446 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:42.759163 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:42.759168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:42.759171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:42.759569 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:42.759794 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:42.760182 (+ 388us) server_negotiation.cc:299] Negotiation successful
0806 02:45:42.760386 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260806 02:45:42.761452 5524 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58136
I20260806 02:45:42.762620 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 5585
I20260806 02:45:42.777172 5990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:42.763578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47034 (local address 127.1.210.65:37959)
0806 02:45:42.763752 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:42.763756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:42.764099 (+ 343us) server_negotiation.cc:407] Connection header received
0806 02:45:42.764224 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:42.764228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:42.764327 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:42.764414 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:42.765774 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:42.766552 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:42.768113 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:42.768680 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:42.768799 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:42.771530 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:42.771535 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:42.771539 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:42.771595 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:42.774395 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:42.775338 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:42.775343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:42.775345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:42.775486 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:42.775882 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:42.775885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:42.775887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:42.776177 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:42.776362 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:42.776731 (+ 369us) server_negotiation.cc:299] Negotiation successful
0806 02:45:42.776916 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:45:42.777927 5708 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:45:42.867019 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 5585
I20260806 02:45:42.884261 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 5719
Aug 06 02:45:42 dist-test-slave-8tdl krb5kdc[5479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.66@KRBTEST.COM
I20260806 02:45:42.902343 5995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:42.885248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37432 (local address 127.1.210.66:32909)
0806 02:45:42.885817 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:42.885826 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:42.885853 (+ 27us) server_negotiation.cc:407] Connection header received
0806 02:45:42.885921 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:42.885927 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:42.886142 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:42.886278 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:42.887884 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:42.889095 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:42.890613 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:42.891409 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:42.891522 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:42.896312 (+ 4790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:42.896326 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:42.896344 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:42.896394 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:42.899360 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:42.900111 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:42.900119 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:42.900121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:42.900290 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:42.900642 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:42.900645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:42.900647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:42.901026 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:42.901268 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:42.901670 (+ 402us) server_negotiation.cc:299] Negotiation successful
0806 02:45:42.901946 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":279,"threads_started":1}
I20260806 02:45:42.903858 5842 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:45:42.992363 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 5719
I20260806 02:45:43.011090 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 5853
Aug 06 02:45:43 dist-test-slave-8tdl krb5kdc[5479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.67@KRBTEST.COM
I20260806 02:45:43.030319 5997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:43.012367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33368 (local address 127.1.210.67:33205)
0806 02:45:43.013073 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:43.013081 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:43.013111 (+ 30us) server_negotiation.cc:407] Connection header received
0806 02:45:43.013175 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:43.013182 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:43.013402 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:43.013638 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:43.015141 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:43.016244 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:43.017727 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:43.018339 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:43.018542 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:43.023645 (+ 5103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:43.023662 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:43.023679 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:43.023728 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:43.027098 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:43.028042 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:43.028052 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:43.028054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:43.028222 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:43.028603 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:43.028606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:43.028609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:43.029085 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:43.029344 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:43.029700 (+ 356us) server_negotiation.cc:299] Negotiation successful
0806 02:45:43.029997 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":379,"threads_started":1}
I20260806 02:45:43.031809 5977 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:45:43.118600 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 5853
I20260806 02:45:43.135800 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 5492
I20260806 02:45:43.137192 5554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:45:43.248772 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 5492
2026-08-06T02:45:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4270 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:45:43 dist-test-slave-8tdl krb5kdc[6002](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:45:43 dist-test-slave-8tdl krb5kdc[6002](info): set up 2 sockets
Aug 06 02:45:43 dist-test-slave-8tdl krb5kdc[6002](info): commencing operation
krb5kdc: starting...
W20260806 02:45:44.446632 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.151s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:45:44 dist-test-slave-8tdl krb5kdc[6002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:45:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
I20260806 02:45:44.691717 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:43929
--webserver_interface=127.1.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:33253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:43929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:45.040431 6015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:45.040769 6015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:45.040927 6015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:45.048938 6015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:45.049069 6015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:45.049147 6015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:45.049249 6015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:45.049312 6015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:45:45.059868 6015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:33253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:43929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:43929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:45.061305 6015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:45.062712 6015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:45.075088 6021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:45.075203 6020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:45.075173 6023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:45.076100 6015 server_base.cc:1034] running on GCE node
I20260806 02:45:45.077108 6015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:45.078848 6015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:45.080113 6015 hybrid_clock.cc:648] HybridClock initialized: now 1785984345080052 us; error 75 us; skew 500 ppm
I20260806 02:45:45.080508 6015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:45:45 dist-test-slave-8tdl krb5kdc[6002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984345, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:45.084111 6015 init.cc:377] Logged in from keytab as kudu/127.1.210.126@KRBTEST.COM (short username kudu)
I20260806 02:45:45.085405 6015 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:45.085541 6015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:45.085867 6015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:45.089097 6015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "bd6fc2a5cfb84a0992b59402e78d3020"
format_stamp: "Formatted at 2026-08-06 02:45:45 on dist-test-slave-8tdl"
server_key: "7acaa519d54ef7a4eecb3a51cc65f398"
server_key_iv: "5544cdce0d7bb6d6966f2682a00dbcee"
server_key_version: "encryptionkey@0"
I20260806 02:45:45.089963 6015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "bd6fc2a5cfb84a0992b59402e78d3020"
format_stamp: "Formatted at 2026-08-06 02:45:45 on dist-test-slave-8tdl"
server_key: "7acaa519d54ef7a4eecb3a51cc65f398"
server_key_iv: "5544cdce0d7bb6d6966f2682a00dbcee"
server_key_version: "encryptionkey@0"
I20260806 02:45:45.096661 6015 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260806 02:45:45.101047 6029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:45.102824 6015 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260806 02:45:45.103050 6015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "bd6fc2a5cfb84a0992b59402e78d3020"
format_stamp: "Formatted at 2026-08-06 02:45:45 on dist-test-slave-8tdl"
server_key: "7acaa519d54ef7a4eecb3a51cc65f398"
server_key_iv: "5544cdce0d7bb6d6966f2682a00dbcee"
server_key_version: "encryptionkey@0"
I20260806 02:45:45.103320 6015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:45.121840 6015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:45.126762 6015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:45.127075 6015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:45.147795 6015 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:43929
I20260806 02:45:45.147882 6080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:43929 every 8 connection(s)
I20260806 02:45:45.149004 6015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:45:45.153311 6081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:45.155614 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 6015
I20260806 02:45:45.155881 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:45:45.156642 1865 external_mini_cluster.cc:1428] Setting key 50e08f33ff64dd8ec4e1107be64fd9b2
I20260806 02:45:45.167663 6081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd6fc2a5cfb84a0992b59402e78d3020: Bootstrap starting.
I20260806 02:45:45.172040 6081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd6fc2a5cfb84a0992b59402e78d3020: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:45.173300 6081 log.cc:824] T 00000000000000000000000000000000 P bd6fc2a5cfb84a0992b59402e78d3020: Log is configured to *not* fsync() on all Append() calls
Aug 06 02:45:45 dist-test-slave-8tdl krb5kdc[6002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:45.177313 6081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd6fc2a5cfb84a0992b59402e78d3020: No bootstrap required, opened a new log
I20260806 02:45:45.185994 6084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:45.159265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48504 (local address 127.1.210.126:43929)
0806 02:45:45.160325 (+ 1060us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:45.160342 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:45.160398 (+ 56us) server_negotiation.cc:407] Connection header received
0806 02:45:45.162902 (+ 2504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:45.162938 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:45.163563 (+ 625us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:45.164095 (+ 532us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:45.165857 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:45.167937 (+ 2080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:45.169567 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:45.170778 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:45.171116 (+ 338us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:45.175555 (+ 4439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:45.175568 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:45.175592 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:45.175647 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:45.180167 (+ 4520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:45.181138 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:45.181148 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:45.181161 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:45.181412 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:45.181900 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:45.181906 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:45.181909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:45.182804 (+ 895us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:45.183121 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:45.183649 (+ 528us) server_negotiation.cc:299] Negotiation successful
0806 02:45:45.184198 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":777,"thread_start_us":434,"threads_started":1}
I20260806 02:45:45.188651 6081 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd6fc2a5cfb84a0992b59402e78d3020 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd6fc2a5cfb84a0992b59402e78d3020" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 43929 } }
I20260806 02:45:45.189075 6081 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd6fc2a5cfb84a0992b59402e78d3020 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:45.189209 6081 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd6fc2a5cfb84a0992b59402e78d3020 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd6fc2a5cfb84a0992b59402e78d3020, State: Initialized, Role: FOLLOWER
I20260806 02:45:45.189824 6081 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd6fc2a5cfb84a0992b59402e78d3020 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd6fc2a5cfb84a0992b59402e78d3020" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 43929 } }
I20260806 02:45:45.190091 6081 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd6fc2a5cfb84a0992b59402e78d3020 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:45:45.190243 6081 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd6fc2a5cfb84a0992b59402e78d3020 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:45:45.190423 6081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd6fc2a5cfb84a0992b59402e78d3020 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:45.192983 6081 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd6fc2a5cfb84a0992b59402e78d3020 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd6fc2a5cfb84a0992b59402e78d3020" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 43929 } }
I20260806 02:45:45.193553 6081 leader_election.cc:302] T 00000000000000000000000000000000 P bd6fc2a5cfb84a0992b59402e78d3020 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd6fc2a5cfb84a0992b59402e78d3020; no voters:
I20260806 02:45:45.194199 6081 leader_election.cc:288] T 00000000000000000000000000000000 P bd6fc2a5cfb84a0992b59402e78d3020 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:45:45.194376 6086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd6fc2a5cfb84a0992b59402e78d3020 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:45.195135 6086 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd6fc2a5cfb84a0992b59402e78d3020 [term 1 LEADER]: Becoming Leader. State: Replica: bd6fc2a5cfb84a0992b59402e78d3020, State: Running, Role: LEADER
I20260806 02:45:45.195696 6086 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd6fc2a5cfb84a0992b59402e78d3020 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd6fc2a5cfb84a0992b59402e78d3020" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 43929 } }
I20260806 02:45:45.196540 6081 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd6fc2a5cfb84a0992b59402e78d3020 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:45.199122 6087 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd6fc2a5cfb84a0992b59402e78d3020 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd6fc2a5cfb84a0992b59402e78d3020" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd6fc2a5cfb84a0992b59402e78d3020" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 43929 } } }
I20260806 02:45:45.199091 6088 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd6fc2a5cfb84a0992b59402e78d3020 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd6fc2a5cfb84a0992b59402e78d3020. Latest consensus state: current_term: 1 leader_uuid: "bd6fc2a5cfb84a0992b59402e78d3020" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd6fc2a5cfb84a0992b59402e78d3020" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 43929 } } }
I20260806 02:45:45.199539 6088 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd6fc2a5cfb84a0992b59402e78d3020 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:45.199539 6087 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd6fc2a5cfb84a0992b59402e78d3020 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:45.200609 6094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:45.208879 6094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:45:45.223515 6094 catalog_manager.cc:1347] Generated new cluster ID: 6ab632612e684c2e9f73c3adef764a72
I20260806 02:45:45.223666 6094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:45.237763 6094 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:45:45.239367 6094 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:45.248311 6094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd6fc2a5cfb84a0992b59402e78d3020: Generated new TSK 0
I20260806 02:45:45.249341 6094 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab.
Entry for principal kudu/127.1.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab.
I20260806 02:45:45.316782 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.65:0
--local_ip_for_outbound_sockets=127.1.210.65
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:43929
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:33253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:45.676841 6107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:45.677148 6107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:45.677306 6107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:45.684876 6107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:45.685003 6107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:45.685180 6107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.65
I20260806 02:45:45.695484 6107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:33253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:43929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.65
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:45.697017 6107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:45.698513 6107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:45.713173 6112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:45.713151 6115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:45.713153 6113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:45.719780 6107 server_base.cc:1034] running on GCE node
I20260806 02:45:45.720746 6107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:45.722483 6107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:45.723765 6107 hybrid_clock.cc:648] HybridClock initialized: now 1785984345723617 us; error 134 us; skew 500 ppm
I20260806 02:45:45.724287 6107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:45:45 dist-test-slave-8tdl krb5kdc[6002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984345, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:45.729146 6107 init.cc:377] Logged in from keytab as kudu/127.1.210.65@KRBTEST.COM (short username kudu)
I20260806 02:45:45.730813 6107 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:45.730949 6107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:45.731317 6107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:45.734508 6107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-0/data/instance:
uuid: "c62ed1fb3d5945fa9d1a6c1177a13037"
format_stamp: "Formatted at 2026-08-06 02:45:45 on dist-test-slave-8tdl"
server_key: "e07aec416a8e6cc8944c39cbaecc22fa"
server_key_iv: "ce0952e0279f80545a3c132def7432e5"
server_key_version: "encryptionkey@0"
I20260806 02:45:45.735471 6107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance:
uuid: "c62ed1fb3d5945fa9d1a6c1177a13037"
format_stamp: "Formatted at 2026-08-06 02:45:45 on dist-test-slave-8tdl"
server_key: "e07aec416a8e6cc8944c39cbaecc22fa"
server_key_iv: "ce0952e0279f80545a3c132def7432e5"
server_key_version: "encryptionkey@0"
I20260806 02:45:45.741743 6107 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260806 02:45:45.746207 6121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:45.748342 6107 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:45:45.748600 6107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-0/wal
uuid: "c62ed1fb3d5945fa9d1a6c1177a13037"
format_stamp: "Formatted at 2026-08-06 02:45:45 on dist-test-slave-8tdl"
server_key: "e07aec416a8e6cc8944c39cbaecc22fa"
server_key_iv: "ce0952e0279f80545a3c132def7432e5"
server_key_version: "encryptionkey@0"
I20260806 02:45:45.748867 6107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:45.767805 6107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:45.772683 6107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:45.772989 6107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:45.774044 6107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:45.775556 6107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:45.775692 6107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:45.775817 6107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:45.775910 6107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:45.811676 6107 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.65:43815
I20260806 02:45:45.811729 6233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.65:43815 every 8 connection(s)
I20260806 02:45:45.812623 6107 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
I20260806 02:45:45.813841 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 6107
I20260806 02:45:45.814079 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance
I20260806 02:45:45.814944 1865 external_mini_cluster.cc:1428] Setting key ca50c66b40a446e2be6613e184e608d0
Aug 06 02:45:45 dist-test-slave-8tdl krb5kdc[6002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984345, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:45.835032 6236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:45.815759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:55823 (local address 127.1.210.126:43929)
0806 02:45:45.816229 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:45.816235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:45.817416 (+ 1181us) server_negotiation.cc:407] Connection header received
0806 02:45:45.819368 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:45.819374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:45.819461 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:45.819577 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:45.821135 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:45.821889 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:45.823798 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:45.824241 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:45.824357 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:45.828624 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:45.828631 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:45.828635 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:45.828682 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:45.831343 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:45.832244 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:45.832253 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:45.832260 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:45.832424 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:45.832940 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:45.832949 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:45.832955 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:45.833402 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:45.833580 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:45.834307 (+ 727us) server_negotiation.cc:299] Negotiation successful
0806 02:45:45.834562 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":212,"threads_started":1}
I20260806 02:45:45.835845 6237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:45.816409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:43929 (local address 127.1.210.65:55823)
0806 02:45:45.817268 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:45.817317 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:45.819098 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:45.819819 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:45.819827 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:45.820366 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:45.820961 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:45.820979 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:45.822054 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:45.822059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:45.823603 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:45.823621 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:45.824494 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:45.824504 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:45.824746 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:45.825453 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:45.825495 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:45.828361 (+ 2866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:45.831521 (+ 3160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:45.831532 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:45.831550 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:45.832064 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:45.832574 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:45.832578 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:45.832580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:45.832798 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:45.833536 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:45.833542 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:45.833881 (+ 339us) client_negotiation.cc:769] Sending connection context
0806 02:45:45.834135 (+ 254us) client_negotiation.cc:240] Negotiation successful
0806 02:45:45.834558 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":340,"threads_started":1}
WARNING: no policy specified for kudu/127.1.210.66@KRBTEST.COM; defaulting to no policy
I20260806 02:45:45.837563 6234 heartbeater.cc:347] Connected to a master server at 127.1.210.126:43929
I20260806 02:45:45.837910 6234 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:45.838639 6234 heartbeater.cc:511] Master 127.1.210.126:43929 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.66@KRBTEST.COM" created.
I20260806 02:45:45.841007 6046 ts_manager.cc:194] Registered new tserver with Master: c62ed1fb3d5945fa9d1a6c1177a13037 (127.1.210.65:43815)
I20260806 02:45:45.842500 6046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.65@KRBTEST.COM'} at 127.1.210.65:55823
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab.
Entry for principal kudu/127.1.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab.
I20260806 02:45:45.871460 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.66:0
--local_ip_for_outbound_sockets=127.1.210.66
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:43929
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:33253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:46.223668 6240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:46.223974 6240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:46.224135 6240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:46.231617 6240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:46.231746 6240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:46.231926 6240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.66
I20260806 02:45:46.241822 6240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:33253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:43929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.66
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:46.243217 6240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:46.244570 6240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:46.258029 6245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:46.258090 6246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:46.258257 6248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:46.259750 6240 server_base.cc:1034] running on GCE node
I20260806 02:45:46.260491 6240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:46.261584 6240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:46.262810 6240 hybrid_clock.cc:648] HybridClock initialized: now 1785984346262752 us; error 48 us; skew 500 ppm
I20260806 02:45:46.263157 6240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:45:46 dist-test-slave-8tdl krb5kdc[6002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984346, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:46.266623 6240 init.cc:377] Logged in from keytab as kudu/127.1.210.66@KRBTEST.COM (short username kudu)
I20260806 02:45:46.267853 6240 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:46.267946 6240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:46.268215 6240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:46.271727 6240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-1/data/instance:
uuid: "7f543bbc44e344b886964e307100e990"
format_stamp: "Formatted at 2026-08-06 02:45:46 on dist-test-slave-8tdl"
server_key: "93cfb8c886e78a8226269d7367891b19"
server_key_iv: "da4c14c096be3c8638fb4f1327c08b0c"
server_key_version: "encryptionkey@0"
I20260806 02:45:46.272538 6240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance:
uuid: "7f543bbc44e344b886964e307100e990"
format_stamp: "Formatted at 2026-08-06 02:45:46 on dist-test-slave-8tdl"
server_key: "93cfb8c886e78a8226269d7367891b19"
server_key_iv: "da4c14c096be3c8638fb4f1327c08b0c"
server_key_version: "encryptionkey@0"
I20260806 02:45:46.277837 6240 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 02:45:46.281687 6254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:46.283278 6240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 02:45:46.283422 6240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-1/wal
uuid: "7f543bbc44e344b886964e307100e990"
format_stamp: "Formatted at 2026-08-06 02:45:46 on dist-test-slave-8tdl"
server_key: "93cfb8c886e78a8226269d7367891b19"
server_key_iv: "da4c14c096be3c8638fb4f1327c08b0c"
server_key_version: "encryptionkey@0"
I20260806 02:45:46.283629 6240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:46.312988 6240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:46.317945 6240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:46.318207 6240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:46.319304 6240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:46.320653 6240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:46.320744 6240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:46.320816 6240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:46.320873 6240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:46.356400 6240 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.66:34001
I20260806 02:45:46.356446 6366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.66:34001 every 8 connection(s)
I20260806 02:45:46.357333 6240 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
I20260806 02:45:46.365114 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 6240
I20260806 02:45:46.365356 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance
I20260806 02:45:46.365906 1865 external_mini_cluster.cc:1428] Setting key b9e592e2accda0a80c0cb7594da33133
Aug 06 02:45:46 dist-test-slave-8tdl krb5kdc[6002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984346, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:46.380273 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:46.359872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:50953 (local address 127.1.210.126:43929)
0806 02:45:46.360326 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:46.360331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:46.361407 (+ 1076us) server_negotiation.cc:407] Connection header received
0806 02:45:46.363420 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:46.363426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:46.363506 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:46.363625 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:46.365299 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:46.366381 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:46.368570 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:46.369260 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:46.369416 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:46.373392 (+ 3976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:46.373401 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:46.373409 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:46.373459 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:46.376780 (+ 3321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:46.377800 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:46.377804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:46.377807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:46.377932 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:46.378417 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:46.378420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:46.378422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:46.378759 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:46.378878 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:46.379808 (+ 930us) server_negotiation.cc:299] Negotiation successful
0806 02:45:46.379988 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":221,"threads_started":1}
I20260806 02:45:46.381901 6370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:46.360490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:43929 (local address 127.1.210.66:50953)
0806 02:45:46.361236 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:46.361281 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:46.363096 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:46.363835 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:46.363843 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:46.364470 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:46.365073 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:46.365092 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:46.366655 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:46.366662 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:46.368394 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:46.368408 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:46.369517 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:46.369522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:46.369725 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:46.370488 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:46.370515 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:46.373126 (+ 2611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:46.376939 (+ 3813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:46.376950 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:46.376971 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:46.377668 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:46.378039 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:46.378046 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:46.378052 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:46.378303 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:46.378886 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:46.378898 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:46.379390 (+ 492us) client_negotiation.cc:769] Sending connection context
0806 02:45:46.379727 (+ 337us) client_negotiation.cc:240] Negotiation successful
0806 02:45:46.380337 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":298,"threads_started":1}
I20260806 02:45:46.383482 6367 heartbeater.cc:347] Connected to a master server at 127.1.210.126:43929
I20260806 02:45:46.383848 6367 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:46.384698 6367 heartbeater.cc:511] Master 127.1.210.126:43929 requested a full tablet report, sending...
I20260806 02:45:46.386379 6046 ts_manager.cc:194] Registered new tserver with Master: 7f543bbc44e344b886964e307100e990 (127.1.210.66:34001)
I20260806 02:45:46.387399 6046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.66@KRBTEST.COM'} at 127.1.210.66:50953
WARNING: no policy specified for kudu/127.1.210.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab.
Entry for principal kudu/127.1.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab.
I20260806 02:45:46.422832 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.67:0
--local_ip_for_outbound_sockets=127.1.210.67
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:43929
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:33253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:46.763531 6373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:46.763828 6373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:46.763985 6373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:46.773236 6373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:46.773358 6373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:46.773547 6373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.67
I20260806 02:45:46.785064 6373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:33253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:43929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.67
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:46.786538 6373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:46.787966 6373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:46.801829 6378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:46.801908 6381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:46.801882 6379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:46.803053 6373 server_base.cc:1034] running on GCE node
I20260806 02:45:46.803812 6373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:46.804940 6373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:46.806212 6373 hybrid_clock.cc:648] HybridClock initialized: now 1785984346806156 us; error 50 us; skew 500 ppm
I20260806 02:45:46.806598 6373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:45:46 dist-test-slave-8tdl krb5kdc[6002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984346, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:46.810257 6373 init.cc:377] Logged in from keytab as kudu/127.1.210.67@KRBTEST.COM (short username kudu)
I20260806 02:45:46.811704 6373 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:46.811843 6373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:46.812242 6373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:46.816332 6373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-2/data/instance:
uuid: "dd4002dd32464f3394285d1cb66c6a38"
format_stamp: "Formatted at 2026-08-06 02:45:46 on dist-test-slave-8tdl"
server_key: "d062a32ddaf60639618dd73b22a9d3ac"
server_key_iv: "953edfe0c4eb8fcc69bd90ea024ac950"
server_key_version: "encryptionkey@0"
I20260806 02:45:46.817438 6373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance:
uuid: "dd4002dd32464f3394285d1cb66c6a38"
format_stamp: "Formatted at 2026-08-06 02:45:46 on dist-test-slave-8tdl"
server_key: "d062a32ddaf60639618dd73b22a9d3ac"
server_key_iv: "953edfe0c4eb8fcc69bd90ea024ac950"
server_key_version: "encryptionkey@0"
I20260806 02:45:46.822959 6373 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 02:45:46.828225 6387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:46.832312 6373 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.001s
I20260806 02:45:46.832538 6373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-2/wal
uuid: "dd4002dd32464f3394285d1cb66c6a38"
format_stamp: "Formatted at 2026-08-06 02:45:46 on dist-test-slave-8tdl"
server_key: "d062a32ddaf60639618dd73b22a9d3ac"
server_key_iv: "953edfe0c4eb8fcc69bd90ea024ac950"
server_key_version: "encryptionkey@0"
I20260806 02:45:46.832796 6373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:46.845667 6234 heartbeater.cc:503] Master 127.1.210.126:43929 was elected leader, sending a full tablet report...
I20260806 02:45:46.864573 6373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:46.869968 6373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:46.870270 6373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:46.871400 6373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:46.872852 6373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:46.872983 6373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:46.873109 6373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:46.873191 6373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:46.908547 6373 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.67:36913
I20260806 02:45:46.908635 6499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.67:36913 every 8 connection(s)
I20260806 02:45:46.909623 6373 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
I20260806 02:45:46.915982 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 6373
I20260806 02:45:46.916194 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance
I20260806 02:45:46.916621 1865 external_mini_cluster.cc:1428] Setting key fa488907f0dc2c134ba7fd110883f986
Aug 06 02:45:46 dist-test-slave-8tdl krb5kdc[6002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984346, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:46.932180 6502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:46.912898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:36565 (local address 127.1.210.126:43929)
0806 02:45:46.913438 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:46.913445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:46.914748 (+ 1303us) server_negotiation.cc:407] Connection header received
0806 02:45:46.917231 (+ 2483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:46.917237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:46.917352 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:46.917465 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:46.919870 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:46.920691 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:46.922277 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:46.922787 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:46.922916 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:46.926409 (+ 3493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:46.926415 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:46.926425 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:46.926458 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:46.929018 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:46.929764 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:46.929769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:46.929773 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:46.929895 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:46.930323 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:46.930329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:46.930332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:46.930608 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:46.930800 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:46.931537 (+ 737us) server_negotiation.cc:299] Negotiation successful
0806 02:45:46.931781 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":283,"threads_started":1}
I20260806 02:45:46.933154 6503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:46.913603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:43929 (local address 127.1.210.67:36565)
0806 02:45:46.914542 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:46.914592 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:46.916917 (+ 2325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:46.918010 (+ 1093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:46.918030 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:46.918931 (+ 901us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:46.919661 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:46.919680 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:46.920847 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:46.920853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:46.922131 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:46.922142 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:46.923029 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:46.923036 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:46.923281 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:46.923986 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:46.924011 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:46.926195 (+ 2184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:46.929164 (+ 2969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:46.929174 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:46.929191 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:46.929633 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:46.930004 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:46.930012 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:46.930016 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:46.930203 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:46.930786 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:46.930794 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:46.931145 (+ 351us) client_negotiation.cc:769] Sending connection context
0806 02:45:46.931410 (+ 265us) client_negotiation.cc:240] Negotiation successful
0806 02:45:46.931971 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":370,"threads_started":1}
I20260806 02:45:46.934521 6500 heartbeater.cc:347] Connected to a master server at 127.1.210.126:43929
I20260806 02:45:46.934849 6500 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:46.935556 6500 heartbeater.cc:511] Master 127.1.210.126:43929 requested a full tablet report, sending...
I20260806 02:45:46.937008 6046 ts_manager.cc:194] Registered new tserver with Master: dd4002dd32464f3394285d1cb66c6a38 (127.1.210.67:36913)
I20260806 02:45:46.937983 6046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.67@KRBTEST.COM'} at 127.1.210.67:36565
I20260806 02:45:46.945326 1865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:45:46.964512 6502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:46.950466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48512 (local address 127.1.210.126:43929)
0806 02:45:46.950756 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:46.950763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:46.951308 (+ 545us) server_negotiation.cc:407] Connection header received
0806 02:45:46.951457 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:46.951462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:46.951574 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:46.951705 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:46.953349 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:46.954303 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:46.955786 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:46.956247 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:46.956359 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:46.958901 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:46.958906 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:46.958910 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:46.958947 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:46.961777 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:46.962513 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:46.962517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:46.962520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:46.962709 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:46.963187 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:46.963190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:46.963193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:46.963557 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:46.963762 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:46.964059 (+ 297us) server_negotiation.cc:299] Negotiation successful
0806 02:45:46.964260 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 02:45:46.968959 6046 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:45:46.976027 6046 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:45:47.001368 6516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:46.989639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55508 (local address 127.1.210.67:36913)
0806 02:45:46.990342 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:46.990350 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:46.991925 (+ 1575us) server_negotiation.cc:407] Connection header received
0806 02:45:46.992245 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:46.992253 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:46.992549 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:46.992715 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:46.993607 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:46.995277 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:46.998615 (+ 3338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:47.000100 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:47.000205 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:47.000452 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:47.000782 (+ 330us) server_negotiation.cc:299] Negotiation successful
0806 02:45:47.000993 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":392,"threads_started":1}
I20260806 02:45:47.001408 6517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:46.989087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:36913 (local address 127.0.0.1:55508)
0806 02:45:46.991800 (+ 2713us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:46.991818 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:46.992027 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:46.992857 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:46.992864 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:46.992897 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:46.993402 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:46.993417 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:46.995441 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:46.995458 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:46.998425 (+ 2967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:46.998460 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:47.000294 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:47.000300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:47.000525 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:47.000589 (+ 64us) client_negotiation.cc:769] Sending connection context
0806 02:45:47.000735 (+ 146us) client_negotiation.cc:240] Negotiation successful
0806 02:45:47.000985 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2594,"mutex_wait_us":295,"thread_start_us":2340,"threads_started":1}
I20260806 02:45:47.001700 6515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:46.989691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42870 (local address 127.1.210.66:34001)
0806 02:45:46.990785 (+ 1094us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:46.990796 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:46.991978 (+ 1182us) server_negotiation.cc:407] Connection header received
0806 02:45:46.992271 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:46.992278 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:46.992530 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:46.992707 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:46.994847 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:46.996429 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:46.998864 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:47.000186 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:47.000333 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:47.000524 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:47.001165 (+ 641us) server_negotiation.cc:299] Negotiation successful
0806 02:45:47.001354 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":910,"thread_start_us":799,"threads_started":1}
I20260806 02:45:47.002099 6518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:46.990879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:34001 (local address 127.0.0.1:42870)
0806 02:45:46.991872 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:46.991888 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:46.992082 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:46.992995 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:46.992998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:46.993055 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:46.993720 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:46.993734 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:46.996571 (+ 2837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:46.996579 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:46.998612 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:46.998627 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:47.000420 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:47.000429 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:47.000606 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:47.000649 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 02:45:47.000744 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 02:45:47.000972 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":871,"mutex_wait_us":75,"spinlock_wait_cycles":2688,"thread_start_us":1398,"threads_started":1}
I20260806 02:45:47.005283 6514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:46.989029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37784 (local address 127.1.210.65:43815)
0806 02:45:46.989845 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:46.989856 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:46.992163 (+ 2307us) server_negotiation.cc:407] Connection header received
0806 02:45:46.992269 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:46.992277 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:46.992530 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:46.992711 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:46.993931 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:46.995600 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:47.000568 (+ 4968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:47.003867 (+ 3299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:47.004037 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:47.004290 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:47.004867 (+ 577us) server_negotiation.cc:299] Negotiation successful
0806 02:45:47.005021 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":472,"threads_started":1}
I20260806 02:45:47.005366 6513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:46.988633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:43815 (local address 127.0.0.1:37784)
0806 02:45:46.991590 (+ 2957us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:46.991646 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:46.991888 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:46.992977 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:46.992993 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:46.993050 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:46.993720 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:46.993734 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:46.995763 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:46.995768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:46.998428 (+ 2660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:46.998444 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:47.004476 (+ 6032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:47.004483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:47.004671 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:47.004709 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 02:45:47.004815 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 02:45:47.004966 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1897,"mutex_wait_us":123,"spinlock_wait_cycles":31360,"thread_start_us":1079,"threads_started":1}
I20260806 02:45:47.006177 6435 tablet_service.cc:1467] Processing CreateTablet for tablet ddb128d0ae0a4ed8b79c2de81cbc99a6 (DEFAULT_TABLE table=test-table [id=1c64ccf7376d4ef2bd28a4b886e6c008]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:47.007730 6435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddb128d0ae0a4ed8b79c2de81cbc99a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:47.009629 6302 tablet_service.cc:1467] Processing CreateTablet for tablet ddb128d0ae0a4ed8b79c2de81cbc99a6 (DEFAULT_TABLE table=test-table [id=1c64ccf7376d4ef2bd28a4b886e6c008]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:47.011086 6302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddb128d0ae0a4ed8b79c2de81cbc99a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:47.019029 6169 tablet_service.cc:1467] Processing CreateTablet for tablet ddb128d0ae0a4ed8b79c2de81cbc99a6 (DEFAULT_TABLE table=test-table [id=1c64ccf7376d4ef2bd28a4b886e6c008]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:47.019999 6519 tablet_bootstrap.cc:492] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P dd4002dd32464f3394285d1cb66c6a38: Bootstrap starting.
I20260806 02:45:47.020614 6169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddb128d0ae0a4ed8b79c2de81cbc99a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:47.023614 6519 tablet_bootstrap.cc:654] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P dd4002dd32464f3394285d1cb66c6a38: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:47.024708 6519 log.cc:824] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P dd4002dd32464f3394285d1cb66c6a38: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:47.025549 6520 tablet_bootstrap.cc:492] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P 7f543bbc44e344b886964e307100e990: Bootstrap starting.
I20260806 02:45:47.029381 6519 tablet_bootstrap.cc:492] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P dd4002dd32464f3394285d1cb66c6a38: No bootstrap required, opened a new log
I20260806 02:45:47.029511 6520 tablet_bootstrap.cc:654] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P 7f543bbc44e344b886964e307100e990: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:47.029695 6519 ts_tablet_manager.cc:1397] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P dd4002dd32464f3394285d1cb66c6a38: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260806 02:45:47.030871 6520 log.cc:824] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P 7f543bbc44e344b886964e307100e990: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:47.034585 6523 tablet_bootstrap.cc:492] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P c62ed1fb3d5945fa9d1a6c1177a13037: Bootstrap starting.
I20260806 02:45:47.038335 6523 tablet_bootstrap.cc:654] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P c62ed1fb3d5945fa9d1a6c1177a13037: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:47.038621 6520 tablet_bootstrap.cc:492] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P 7f543bbc44e344b886964e307100e990: No bootstrap required, opened a new log
I20260806 02:45:47.038928 6520 ts_tablet_manager.cc:1397] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P 7f543bbc44e344b886964e307100e990: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.006s
I20260806 02:45:47.039691 6523 log.cc:824] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P c62ed1fb3d5945fa9d1a6c1177a13037: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:47.041658 6519 raft_consensus.cc:348] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P dd4002dd32464f3394285d1cb66c6a38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c62ed1fb3d5945fa9d1a6c1177a13037" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 43815 } } peers { permanent_uuid: "dd4002dd32464f3394285d1cb66c6a38" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 36913 } } peers { permanent_uuid: "7f543bbc44e344b886964e307100e990" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 34001 } }
I20260806 02:45:47.042131 6519 raft_consensus.cc:374] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P dd4002dd32464f3394285d1cb66c6a38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:47.042222 6519 raft_consensus.cc:729] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P dd4002dd32464f3394285d1cb66c6a38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd4002dd32464f3394285d1cb66c6a38, State: Initialized, Role: FOLLOWER
I20260806 02:45:47.042655 6523 tablet_bootstrap.cc:492] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P c62ed1fb3d5945fa9d1a6c1177a13037: No bootstrap required, opened a new log
I20260806 02:45:47.042968 6523 ts_tablet_manager.cc:1397] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P c62ed1fb3d5945fa9d1a6c1177a13037: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260806 02:45:47.043040 6519 consensus_queue.cc:260] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P dd4002dd32464f3394285d1cb66c6a38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c62ed1fb3d5945fa9d1a6c1177a13037" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 43815 } } peers { permanent_uuid: "dd4002dd32464f3394285d1cb66c6a38" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 36913 } } peers { permanent_uuid: "7f543bbc44e344b886964e307100e990" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 34001 } }
I20260806 02:45:47.047344 6500 heartbeater.cc:503] Master 127.1.210.126:43929 was elected leader, sending a full tablet report...
I20260806 02:45:47.048077 6519 ts_tablet_manager.cc:1428] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P dd4002dd32464f3394285d1cb66c6a38: Time spent starting tablet: real 0.018s user 0.015s sys 0.004s
I20260806 02:45:47.050731 6520 raft_consensus.cc:348] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P 7f543bbc44e344b886964e307100e990 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c62ed1fb3d5945fa9d1a6c1177a13037" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 43815 } } peers { permanent_uuid: "dd4002dd32464f3394285d1cb66c6a38" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 36913 } } peers { permanent_uuid: "7f543bbc44e344b886964e307100e990" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 34001 } }
I20260806 02:45:47.051257 6520 raft_consensus.cc:374] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P 7f543bbc44e344b886964e307100e990 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:47.051388 6520 raft_consensus.cc:729] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P 7f543bbc44e344b886964e307100e990 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f543bbc44e344b886964e307100e990, State: Initialized, Role: FOLLOWER
I20260806 02:45:47.051291 6523 raft_consensus.cc:348] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P c62ed1fb3d5945fa9d1a6c1177a13037 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c62ed1fb3d5945fa9d1a6c1177a13037" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 43815 } } peers { permanent_uuid: "dd4002dd32464f3394285d1cb66c6a38" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 36913 } } peers { permanent_uuid: "7f543bbc44e344b886964e307100e990" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 34001 } }
I20260806 02:45:47.051764 6523 raft_consensus.cc:374] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P c62ed1fb3d5945fa9d1a6c1177a13037 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:47.051872 6523 raft_consensus.cc:729] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P c62ed1fb3d5945fa9d1a6c1177a13037 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c62ed1fb3d5945fa9d1a6c1177a13037, State: Initialized, Role: FOLLOWER
I20260806 02:45:47.052197 6520 consensus_queue.cc:260] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P 7f543bbc44e344b886964e307100e990 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c62ed1fb3d5945fa9d1a6c1177a13037" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 43815 } } peers { permanent_uuid: "dd4002dd32464f3394285d1cb66c6a38" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 36913 } } peers { permanent_uuid: "7f543bbc44e344b886964e307100e990" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 34001 } }
I20260806 02:45:47.052484 6523 consensus_queue.cc:260] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P c62ed1fb3d5945fa9d1a6c1177a13037 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c62ed1fb3d5945fa9d1a6c1177a13037" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 43815 } } peers { permanent_uuid: "dd4002dd32464f3394285d1cb66c6a38" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 36913 } } peers { permanent_uuid: "7f543bbc44e344b886964e307100e990" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 34001 } }
I20260806 02:45:47.053740 6367 heartbeater.cc:503] Master 127.1.210.126:43929 was elected leader, sending a full tablet report...
I20260806 02:45:47.054189 6520 ts_tablet_manager.cc:1428] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P 7f543bbc44e344b886964e307100e990: Time spent starting tablet: real 0.015s user 0.004s sys 0.010s
I20260806 02:45:47.056241 6523 ts_tablet_manager.cc:1428] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P c62ed1fb3d5945fa9d1a6c1177a13037: Time spent starting tablet: real 0.013s user 0.010s sys 0.002s
W20260806 02:45:47.065171 6235 tablet.cc:2369] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P c62ed1fb3d5945fa9d1a6c1177a13037: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:45:47.109345 6368 tablet.cc:2369] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P 7f543bbc44e344b886964e307100e990: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:45:47.161607 6501 tablet.cc:2369] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P dd4002dd32464f3394285d1cb66c6a38: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:45:47.256896 6527 raft_consensus.cc:482] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P c62ed1fb3d5945fa9d1a6c1177a13037 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:45:47.257179 6527 raft_consensus.cc:504] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P c62ed1fb3d5945fa9d1a6c1177a13037 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c62ed1fb3d5945fa9d1a6c1177a13037" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 43815 } } peers { permanent_uuid: "dd4002dd32464f3394285d1cb66c6a38" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 36913 } } peers { permanent_uuid: "7f543bbc44e344b886964e307100e990" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 34001 } }
I20260806 02:45:47.258929 6527 leader_election.cc:288] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P c62ed1fb3d5945fa9d1a6c1177a13037 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd4002dd32464f3394285d1cb66c6a38 (127.1.210.67:36913), 7f543bbc44e344b886964e307100e990 (127.1.210.66:34001)
I20260806 02:45:47.265692 6516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:47.259487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:56219 (local address 127.1.210.67:36913)
0806 02:45:47.259762 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:47.259772 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:47.260308 (+ 536us) server_negotiation.cc:407] Connection header received
0806 02:45:47.260522 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:47.260526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:47.260590 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:47.260703 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:47.261259 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:47.262009 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:47.263946 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:47.264727 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:47.264912 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:47.265006 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:47.265297 (+ 291us) server_negotiation.cc:299] Negotiation successful
0806 02:45:47.265423 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 02:45:47.265724 6515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:47.259507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:50745 (local address 127.1.210.66:34001)
0806 02:45:47.260056 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:47.260064 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:47.260614 (+ 550us) server_negotiation.cc:407] Connection header received
0806 02:45:47.260794 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:47.260798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:47.260855 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:47.260955 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:47.261539 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:47.262254 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:47.264023 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:47.264785 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:47.264980 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:47.265057 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:47.265328 (+ 271us) server_negotiation.cc:299] Negotiation successful
0806 02:45:47.265454 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298}
I20260806 02:45:47.265878 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:47.259240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:34001 (local address 127.1.210.65:50745)
0806 02:45:47.260521 (+ 1281us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:47.260537 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:47.260658 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:47.260964 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:47.260969 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:47.260998 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:47.261387 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:47.261395 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:47.262396 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:47.262401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:47.263863 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:47.263876 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:47.265024 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:47.265030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:47.265179 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:47.265202 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 02:45:47.265301 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 02:45:47.265429 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1152,"mutex_wait_us":27,"thread_start_us":543,"threads_started":1}
I20260806 02:45:47.266090 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:47.259152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:36913 (local address 127.1.210.65:56219)
0806 02:45:47.260166 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:47.260190 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:47.260377 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:47.260723 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:47.260728 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:47.260755 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:47.261114 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:47.261124 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:47.262140 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:47.262146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:47.263778 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:47.263795 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:47.264968 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:47.264976 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:47.265136 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:47.265161 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 02:45:47.265262 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 02:45:47.265398 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"mutex_wait_us":3,"thread_start_us":1033,"threads_started":1}
I20260806 02:45:47.266937 6322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb128d0ae0a4ed8b79c2de81cbc99a6" candidate_uuid: "c62ed1fb3d5945fa9d1a6c1177a13037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f543bbc44e344b886964e307100e990" is_pre_election: true
I20260806 02:45:47.267032 6455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb128d0ae0a4ed8b79c2de81cbc99a6" candidate_uuid: "c62ed1fb3d5945fa9d1a6c1177a13037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd4002dd32464f3394285d1cb66c6a38" is_pre_election: true
I20260806 02:45:47.267480 6322 raft_consensus.cc:2454] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P 7f543bbc44e344b886964e307100e990 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c62ed1fb3d5945fa9d1a6c1177a13037 in term 0.
I20260806 02:45:47.267536 6455 raft_consensus.cc:2454] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P dd4002dd32464f3394285d1cb66c6a38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c62ed1fb3d5945fa9d1a6c1177a13037 in term 0.
I20260806 02:45:47.268131 6123 leader_election.cc:302] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P c62ed1fb3d5945fa9d1a6c1177a13037 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f543bbc44e344b886964e307100e990, c62ed1fb3d5945fa9d1a6c1177a13037; no voters:
I20260806 02:45:47.268530 6527 raft_consensus.cc:2781] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P c62ed1fb3d5945fa9d1a6c1177a13037 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:45:47.268685 6527 raft_consensus.cc:482] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P c62ed1fb3d5945fa9d1a6c1177a13037 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:45:47.268795 6527 raft_consensus.cc:3037] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P c62ed1fb3d5945fa9d1a6c1177a13037 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:47.271029 6527 raft_consensus.cc:504] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P c62ed1fb3d5945fa9d1a6c1177a13037 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c62ed1fb3d5945fa9d1a6c1177a13037" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 43815 } } peers { permanent_uuid: "dd4002dd32464f3394285d1cb66c6a38" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 36913 } } peers { permanent_uuid: "7f543bbc44e344b886964e307100e990" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 34001 } }
I20260806 02:45:47.271656 6527 leader_election.cc:288] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P c62ed1fb3d5945fa9d1a6c1177a13037 [CANDIDATE]: Term 1 election: Requested vote from peers dd4002dd32464f3394285d1cb66c6a38 (127.1.210.67:36913), 7f543bbc44e344b886964e307100e990 (127.1.210.66:34001)
I20260806 02:45:47.272295 6322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb128d0ae0a4ed8b79c2de81cbc99a6" candidate_uuid: "c62ed1fb3d5945fa9d1a6c1177a13037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f543bbc44e344b886964e307100e990"
I20260806 02:45:47.272296 6455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddb128d0ae0a4ed8b79c2de81cbc99a6" candidate_uuid: "c62ed1fb3d5945fa9d1a6c1177a13037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd4002dd32464f3394285d1cb66c6a38"
I20260806 02:45:47.272571 6455 raft_consensus.cc:3037] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P dd4002dd32464f3394285d1cb66c6a38 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:47.272553 6322 raft_consensus.cc:3037] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P 7f543bbc44e344b886964e307100e990 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:47.276084 6322 raft_consensus.cc:2454] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P 7f543bbc44e344b886964e307100e990 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c62ed1fb3d5945fa9d1a6c1177a13037 in term 1.
I20260806 02:45:47.276084 6455 raft_consensus.cc:2454] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P dd4002dd32464f3394285d1cb66c6a38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c62ed1fb3d5945fa9d1a6c1177a13037 in term 1.
I20260806 02:45:47.276641 6124 leader_election.cc:302] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P c62ed1fb3d5945fa9d1a6c1177a13037 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c62ed1fb3d5945fa9d1a6c1177a13037, dd4002dd32464f3394285d1cb66c6a38; no voters:
I20260806 02:45:47.277001 6527 raft_consensus.cc:2781] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P c62ed1fb3d5945fa9d1a6c1177a13037 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:47.277447 6527 raft_consensus.cc:686] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P c62ed1fb3d5945fa9d1a6c1177a13037 [term 1 LEADER]: Becoming Leader. State: Replica: c62ed1fb3d5945fa9d1a6c1177a13037, State: Running, Role: LEADER
I20260806 02:45:47.278204 6527 consensus_queue.cc:237] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P c62ed1fb3d5945fa9d1a6c1177a13037 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c62ed1fb3d5945fa9d1a6c1177a13037" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 43815 } } peers { permanent_uuid: "dd4002dd32464f3394285d1cb66c6a38" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 36913 } } peers { permanent_uuid: "7f543bbc44e344b886964e307100e990" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 34001 } }
I20260806 02:45:47.282922 6045 catalog_manager.cc:5495] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P c62ed1fb3d5945fa9d1a6c1177a13037 reported cstate change: term changed from 0 to 1, leader changed from <none> to c62ed1fb3d5945fa9d1a6c1177a13037 (127.1.210.65). New cstate: current_term: 1 leader_uuid: "c62ed1fb3d5945fa9d1a6c1177a13037" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c62ed1fb3d5945fa9d1a6c1177a13037" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 43815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd4002dd32464f3394285d1cb66c6a38" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 36913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f543bbc44e344b886964e307100e990" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 34001 } health_report { overall_health: UNKNOWN } } }
I20260806 02:45:47.302971 6514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:47.296711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37794 (local address 127.1.210.65:43815)
0806 02:45:47.296991 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:47.296998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:47.297089 (+ 91us) server_negotiation.cc:407] Connection header received
0806 02:45:47.297376 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:47.297381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:47.297521 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:47.297691 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:47.298177 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:47.299055 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:47.300558 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:47.300957 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:47.301093 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:47.301801 (+ 708us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:47.302365 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:47.302479 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:47.302602 (+ 123us) server_negotiation.cc:299] Negotiation successful
0806 02:45:47.302717 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260806 02:45:47.311769 6455 raft_consensus.cc:1261] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P dd4002dd32464f3394285d1cb66c6a38 [term 1 FOLLOWER]: Refusing update from remote peer c62ed1fb3d5945fa9d1a6c1177a13037: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:47.311772 6322 raft_consensus.cc:1261] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P 7f543bbc44e344b886964e307100e990 [term 1 FOLLOWER]: Refusing update from remote peer c62ed1fb3d5945fa9d1a6c1177a13037: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:47.312858 6527 consensus_queue.cc:1035] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P c62ed1fb3d5945fa9d1a6c1177a13037 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f543bbc44e344b886964e307100e990" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 34001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:47.313225 6530 consensus_queue.cc:1035] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P c62ed1fb3d5945fa9d1a6c1177a13037 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd4002dd32464f3394285d1cb66c6a38" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 36913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:47.325392 6533 mvcc.cc:205] Tried to move back new op lower bound from 7315391886578061312 to 7315391886455914496. Current Snapshot: MvccSnapshot[applied={T|T < 7315391886578061312}]
I20260806 02:45:47.327641 6535 mvcc.cc:205] Tried to move back new op lower bound from 7315391886578061312 to 7315391886455914496. Current Snapshot: MvccSnapshot[applied={T|T < 7315391886578061312}]
I20260806 02:45:47.341760 6046 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:47.342087 6046 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:47.345721 6046 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd6fc2a5cfb84a0992b59402e78d3020: Sending DeleteTablet for 3 replicas of tablet ddb128d0ae0a4ed8b79c2de81cbc99a6
I20260806 02:45:47.347187 6302 tablet_service.cc:1514] Processing DeleteTablet for tablet ddb128d0ae0a4ed8b79c2de81cbc99a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:47 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:42870
I20260806 02:45:47.347208 6435 tablet_service.cc:1514] Processing DeleteTablet for tablet ddb128d0ae0a4ed8b79c2de81cbc99a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:47 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:55508
I20260806 02:45:47.347378 6169 tablet_service.cc:1514] Processing DeleteTablet for tablet ddb128d0ae0a4ed8b79c2de81cbc99a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:47 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:37784
I20260806 02:45:47.349725 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 6107
I20260806 02:45:47.357366 6545 tablet_replica.cc:330] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P c62ed1fb3d5945fa9d1a6c1177a13037: stopping tablet replica
I20260806 02:45:47.358157 6543 tablet_replica.cc:330] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P dd4002dd32464f3394285d1cb66c6a38: stopping tablet replica
I20260806 02:45:47.358733 6545 raft_consensus.cc:2229] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P c62ed1fb3d5945fa9d1a6c1177a13037 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:45:47.359472 6545 raft_consensus.cc:2258] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P c62ed1fb3d5945fa9d1a6c1177a13037 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:47.359968 6543 raft_consensus.cc:2229] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P dd4002dd32464f3394285d1cb66c6a38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:47.361059 6543 raft_consensus.cc:2258] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P dd4002dd32464f3394285d1cb66c6a38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:47.363638 6543 ts_tablet_manager.cc:1905] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P dd4002dd32464f3394285d1cb66c6a38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:47.364292 6544 tablet_replica.cc:330] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P 7f543bbc44e344b886964e307100e990: stopping tablet replica
I20260806 02:45:47.364696 6544 raft_consensus.cc:2229] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P 7f543bbc44e344b886964e307100e990 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:47.365211 6544 raft_consensus.cc:2258] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P 7f543bbc44e344b886964e307100e990 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:47.372450 6545 ts_tablet_manager.cc:1905] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P c62ed1fb3d5945fa9d1a6c1177a13037: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:47.372423 6544 ts_tablet_manager.cc:1905] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P 7f543bbc44e344b886964e307100e990: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:47.379874 6544 ts_tablet_manager.cc:1918] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P 7f543bbc44e344b886964e307100e990: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:45:47.379954 6545 ts_tablet_manager.cc:1918] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P c62ed1fb3d5945fa9d1a6c1177a13037: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:45:47.380093 6544 log.cc:1196] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P 7f543bbc44e344b886964e307100e990: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-1/wal/wals/ddb128d0ae0a4ed8b79c2de81cbc99a6
I20260806 02:45:47.380102 6545 log.cc:1196] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P c62ed1fb3d5945fa9d1a6c1177a13037: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-0/wal/wals/ddb128d0ae0a4ed8b79c2de81cbc99a6
I20260806 02:45:47.380692 6544 ts_tablet_manager.cc:1939] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P 7f543bbc44e344b886964e307100e990: Deleting consensus metadata
I20260806 02:45:47.380712 6545 ts_tablet_manager.cc:1939] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P c62ed1fb3d5945fa9d1a6c1177a13037: Deleting consensus metadata
I20260806 02:45:47.380787 6543 ts_tablet_manager.cc:1918] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P dd4002dd32464f3394285d1cb66c6a38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:45:47.380996 6543 log.cc:1196] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P dd4002dd32464f3394285d1cb66c6a38: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984304136196-1865-0/minicluster-data/ts-2/wal/wals/ddb128d0ae0a4ed8b79c2de81cbc99a6
I20260806 02:45:47.381628 6543 ts_tablet_manager.cc:1939] T ddb128d0ae0a4ed8b79c2de81cbc99a6 P dd4002dd32464f3394285d1cb66c6a38: Deleting consensus metadata
I20260806 02:45:47.382659 6031 catalog_manager.cc:4841] TS 7f543bbc44e344b886964e307100e990 (127.1.210.66:34001): tablet ddb128d0ae0a4ed8b79c2de81cbc99a6 (table test-table [id=1c64ccf7376d4ef2bd28a4b886e6c008]) successfully deleted
I20260806 02:45:47.382637 6030 catalog_manager.cc:4841] TS c62ed1fb3d5945fa9d1a6c1177a13037 (127.1.210.65:43815): tablet ddb128d0ae0a4ed8b79c2de81cbc99a6 (table test-table [id=1c64ccf7376d4ef2bd28a4b886e6c008]) successfully deleted
I20260806 02:45:47.383518 6032 catalog_manager.cc:4841] TS dd4002dd32464f3394285d1cb66c6a38 (127.1.210.67:36913): tablet ddb128d0ae0a4ed8b79c2de81cbc99a6 (table test-table [id=1c64ccf7376d4ef2bd28a4b886e6c008]) successfully deleted
Aug 06 02:45:47 dist-test-slave-8tdl krb5kdc[6002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.65@KRBTEST.COM
I20260806 02:45:47.395509 6514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:47.350799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37810 (local address 127.1.210.65:43815)
0806 02:45:47.351040 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:47.351051 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:47.369462 (+ 18411us) server_negotiation.cc:407] Connection header received
0806 02:45:47.369732 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:47.369738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:47.369827 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:47.369916 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:47.372388 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:47.373633 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:47.375644 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:47.376381 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:47.376505 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:47.387890 (+ 11385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:47.387907 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:47.387926 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:47.388019 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:47.392224 (+ 4205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:47.393146 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:47.393162 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:47.393165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:47.393356 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:47.393784 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:47.393789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:47.393793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:47.394256 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:47.394604 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:47.394843 (+ 239us) server_negotiation.cc:299] Negotiation successful
0806 02:45:47.395113 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 02:45:47.396245 6229 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:45:47.496325 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 6107
I20260806 02:45:47.516177 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 6240
Aug 06 02:45:47 dist-test-slave-8tdl krb5kdc[6002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.66@KRBTEST.COM
I20260806 02:45:47.532568 6515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:47.517106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42876 (local address 127.1.210.66:34001)
0806 02:45:47.517317 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:47.517325 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:47.517400 (+ 75us) server_negotiation.cc:407] Connection header received
0806 02:45:47.517659 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:47.517663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:47.517740 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:47.517848 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:47.519227 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:47.519982 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:47.521240 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:47.521663 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:47.521798 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:47.526312 (+ 4514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:47.526328 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:47.526345 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:47.526387 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:47.529460 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:47.530264 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:47.530272 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:47.530274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:47.530417 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:47.530933 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:47.530937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:47.530939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:47.531342 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:47.531665 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:47.531972 (+ 307us) server_negotiation.cc:299] Negotiation successful
0806 02:45:47.532207 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 02:45:47.533296 6362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:45:47.625810 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 6240
I20260806 02:45:47.643163 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 6373
Aug 06 02:45:47 dist-test-slave-8tdl krb5kdc[6002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.67@KRBTEST.COM
I20260806 02:45:47.659773 6516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:47.644013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55512 (local address 127.1.210.67:36913)
0806 02:45:47.644176 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:47.644180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:47.644199 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:45:47.644392 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:47.644398 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:47.644473 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:47.644587 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:47.646029 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:47.646863 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:47.648127 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:47.648539 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:47.648641 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:47.653461 (+ 4820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:47.653478 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:47.653500 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:47.653540 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:47.656553 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:47.657307 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:47.657315 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:47.657317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:47.657491 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:47.657869 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:47.657873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:47.657875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:47.658455 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:47.658779 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:47.659197 (+ 418us) server_negotiation.cc:299] Negotiation successful
0806 02:45:47.659373 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:45:47.660491 6495 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:45:47.756922 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 6373
I20260806 02:45:47.774773 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 6015
I20260806 02:45:47.775980 6076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:45:47.886368 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 6015
2026-08-06T02:45:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4634 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-06T02:45:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:47Z Disabled control of system clock
I20260806 02:45:47.960675 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:39719
--webserver_interface=127.1.210.126
--webserver_port=0
--builtin_ntp_servers=127.1.210.84:43271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:39719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 02:45:48.302498 6554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:48.302893 6554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:48.303053 6554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:48.310531 6554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:48.310648 6554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:48.310731 6554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:48.310822 6554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:48.310883 6554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:45:48.320418 6554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:43271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:39719
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:39719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:48.321878 6554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:48.323870 6554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:48.337656 6562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:48.337671 6559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:48.337672 6560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:48.338809 6554 server_base.cc:1034] running on GCE node
I20260806 02:45:48.339691 6554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:48.341292 6554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:48.342567 6554 hybrid_clock.cc:648] HybridClock initialized: now 1785984348342516 us; error 66 us; skew 500 ppm
I20260806 02:45:48.343011 6554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:45:48.345489 6554 webserver.cc:460] Webserver started at http://127.1.210.126:36913/ using document root <none> and password file <none>
I20260806 02:45:48.346271 6554 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:48.346408 6554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:48.346802 6554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:48.349848 6554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "8a1fea205d6b4a69a630fd5f9680e883"
format_stamp: "Formatted at 2026-08-06 02:45:48 on dist-test-slave-8tdl"
server_key: "bf87361d346dc0a1e4d99f0f43febdeb"
server_key_iv: "1412c367bdeca3655c1c5f0ce0f30427"
server_key_version: "encryptionkey@0"
I20260806 02:45:48.350773 6554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "8a1fea205d6b4a69a630fd5f9680e883"
format_stamp: "Formatted at 2026-08-06 02:45:48 on dist-test-slave-8tdl"
server_key: "bf87361d346dc0a1e4d99f0f43febdeb"
server_key_iv: "1412c367bdeca3655c1c5f0ce0f30427"
server_key_version: "encryptionkey@0"
I20260806 02:45:48.356393 6554 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260806 02:45:48.360294 6568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:48.361806 6554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:45:48.361997 6554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "8a1fea205d6b4a69a630fd5f9680e883"
format_stamp: "Formatted at 2026-08-06 02:45:48 on dist-test-slave-8tdl"
server_key: "bf87361d346dc0a1e4d99f0f43febdeb"
server_key_iv: "1412c367bdeca3655c1c5f0ce0f30427"
server_key_version: "encryptionkey@0"
I20260806 02:45:48.362223 6554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:48.368105 6554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:48.368913 6554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:48.369215 6554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:48.387954 6554 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:39719
I20260806 02:45:48.388025 6619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:39719 every 8 connection(s)
I20260806 02:45:48.389374 6554 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:45:48.392271 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 6554
I20260806 02:45:48.392520 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:45:48.393267 1865 external_mini_cluster.cc:1428] Setting key 95ad1c371e47ea8bcef3b52569d497c1
I20260806 02:45:48.393896 6620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:48.403796 6623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:48.395660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33314 (local address 127.1.210.126:39719)
0806 02:45:48.396855 (+ 1195us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:48.396882 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:48.396941 (+ 59us) server_negotiation.cc:407] Connection header received
0806 02:45:48.399216 (+ 2275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:48.399249 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:48.399879 (+ 630us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:48.400381 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:48.401524 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:48.401532 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:48.401549 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:48.401567 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:48.401673 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:48.401697 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:48.401937 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:48.402151 (+ 214us) server_negotiation.cc:299] Negotiation successful
0806 02:45:48.402476 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":367,"threads_started":1}
I20260806 02:45:48.409199 6620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a1fea205d6b4a69a630fd5f9680e883: Bootstrap starting.
I20260806 02:45:48.412351 6620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a1fea205d6b4a69a630fd5f9680e883: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:48.413393 6620 log.cc:824] T 00000000000000000000000000000000 P 8a1fea205d6b4a69a630fd5f9680e883: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:48.416669 6620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a1fea205d6b4a69a630fd5f9680e883: No bootstrap required, opened a new log
I20260806 02:45:48.424628 6620 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a1fea205d6b4a69a630fd5f9680e883 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a1fea205d6b4a69a630fd5f9680e883" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39719 } }
I20260806 02:45:48.425012 6620 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a1fea205d6b4a69a630fd5f9680e883 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:48.425143 6620 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a1fea205d6b4a69a630fd5f9680e883 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a1fea205d6b4a69a630fd5f9680e883, State: Initialized, Role: FOLLOWER
I20260806 02:45:48.425746 6620 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a1fea205d6b4a69a630fd5f9680e883 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a1fea205d6b4a69a630fd5f9680e883" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39719 } }
I20260806 02:45:48.425978 6620 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a1fea205d6b4a69a630fd5f9680e883 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:45:48.426119 6620 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a1fea205d6b4a69a630fd5f9680e883 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:45:48.426298 6620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a1fea205d6b4a69a630fd5f9680e883 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:48.428891 6620 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a1fea205d6b4a69a630fd5f9680e883 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a1fea205d6b4a69a630fd5f9680e883" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39719 } }
I20260806 02:45:48.429513 6620 leader_election.cc:302] T 00000000000000000000000000000000 P 8a1fea205d6b4a69a630fd5f9680e883 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a1fea205d6b4a69a630fd5f9680e883; no voters:
I20260806 02:45:48.430202 6620 leader_election.cc:288] T 00000000000000000000000000000000 P 8a1fea205d6b4a69a630fd5f9680e883 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:45:48.430335 6625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a1fea205d6b4a69a630fd5f9680e883 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:48.431172 6625 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a1fea205d6b4a69a630fd5f9680e883 [term 1 LEADER]: Becoming Leader. State: Replica: 8a1fea205d6b4a69a630fd5f9680e883, State: Running, Role: LEADER
I20260806 02:45:48.431797 6625 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a1fea205d6b4a69a630fd5f9680e883 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a1fea205d6b4a69a630fd5f9680e883" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39719 } }
I20260806 02:45:48.432474 6620 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a1fea205d6b4a69a630fd5f9680e883 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:48.434999 6627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a1fea205d6b4a69a630fd5f9680e883 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a1fea205d6b4a69a630fd5f9680e883. Latest consensus state: current_term: 1 leader_uuid: "8a1fea205d6b4a69a630fd5f9680e883" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a1fea205d6b4a69a630fd5f9680e883" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39719 } } }
I20260806 02:45:48.435501 6627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a1fea205d6b4a69a630fd5f9680e883 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:48.435844 6626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a1fea205d6b4a69a630fd5f9680e883 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a1fea205d6b4a69a630fd5f9680e883" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a1fea205d6b4a69a630fd5f9680e883" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39719 } } }
I20260806 02:45:48.436317 6626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a1fea205d6b4a69a630fd5f9680e883 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:48.438895 6634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:48.443442 6634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:45:48.452489 6634 catalog_manager.cc:1347] Generated new cluster ID: d1cd7194126d4ca28dc61bd831ec3662
I20260806 02:45:48.452639 6634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:48.463474 6634 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:45:48.463910 6634 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:48.475469 6634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a1fea205d6b4a69a630fd5f9680e883: Generated new TSK 0
I20260806 02:45:48.476687 6634 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:45:48.490998 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.65:0
--local_ip_for_outbound_sockets=127.1.210.65
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:39719
--builtin_ntp_servers=127.1.210.84:43271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 02:45:48.838876 6644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:48.839196 6644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:48.839356 6644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:48.846961 6644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:48.847071 6644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:48.847257 6644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.65
I20260806 02:45:48.856701 6644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:43271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:39719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.65
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:48.858179 6644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:48.860405 6644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:48.875079 6649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:48.874976 6650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:48.875109 6652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:48.875990 6644 server_base.cc:1034] running on GCE node
I20260806 02:45:48.876655 6644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:48.877707 6644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:48.878966 6644 hybrid_clock.cc:648] HybridClock initialized: now 1785984348878863 us; error 65 us; skew 500 ppm
I20260806 02:45:48.879315 6644 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:45:48.881970 6644 webserver.cc:460] Webserver started at http://127.1.210.65:33785/ using document root <none> and password file <none>
I20260806 02:45:48.882786 6644 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:48.882921 6644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:48.883255 6644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:48.886312 6644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-0/data/instance:
uuid: "456d8c4f09124da19f1d4ecf9a319f82"
format_stamp: "Formatted at 2026-08-06 02:45:48 on dist-test-slave-8tdl"
server_key: "a5c3517e592e16186183bf9e4bcf1347"
server_key_iv: "a9b974ad1f153faa7010a64a96a01a65"
server_key_version: "encryptionkey@0"
I20260806 02:45:48.887210 6644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance:
uuid: "456d8c4f09124da19f1d4ecf9a319f82"
format_stamp: "Formatted at 2026-08-06 02:45:48 on dist-test-slave-8tdl"
server_key: "a5c3517e592e16186183bf9e4bcf1347"
server_key_iv: "a9b974ad1f153faa7010a64a96a01a65"
server_key_version: "encryptionkey@0"
I20260806 02:45:48.892808 6644 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260806 02:45:48.896921 6658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:48.898469 6644 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:45:48.898666 6644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-0/wal
uuid: "456d8c4f09124da19f1d4ecf9a319f82"
format_stamp: "Formatted at 2026-08-06 02:45:48 on dist-test-slave-8tdl"
server_key: "a5c3517e592e16186183bf9e4bcf1347"
server_key_iv: "a9b974ad1f153faa7010a64a96a01a65"
server_key_version: "encryptionkey@0"
I20260806 02:45:48.899026 6644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:48.905583 6644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:48.906487 6644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:48.906849 6644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:48.907855 6644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:48.909343 6644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:48.909464 6644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:48.909600 6644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:48.909693 6644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:48.944896 6644 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.65:45265
I20260806 02:45:48.944941 6770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.65:45265 every 8 connection(s)
I20260806 02:45:48.946309 6644 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
I20260806 02:45:48.954039 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 6644
I20260806 02:45:48.954381 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance
I20260806 02:45:48.955366 1865 external_mini_cluster.cc:1428] Setting key 8fe97b5473043c324ba995b461e5396d
I20260806 02:45:48.956148 6773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:48.949085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:34433 (local address 127.1.210.126:39719)
0806 02:45:48.949598 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:48.949604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:48.951191 (+ 1587us) server_negotiation.cc:407] Connection header received
0806 02:45:48.953012 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:48.953017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:48.953077 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:48.953184 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:48.955181 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:48.955186 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:48.955189 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:48.955207 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:48.955254 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:48.955267 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:48.955378 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:48.955752 (+ 374us) server_negotiation.cc:299] Negotiation successful
0806 02:45:48.955838 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":257,"threads_started":1}
I20260806 02:45:48.957476 6774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:48.949765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:39719 (local address 127.1.210.65:34433)
0806 02:45:48.950871 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:48.950913 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:48.952684 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:48.953454 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:48.953467 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:48.953612 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:48.954752 (+ 1140us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:45:48.954756 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:48.954815 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:45:48.954958 (+ 143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:48.955382 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:48.955396 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:48.955405 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 02:45:48.955820 (+ 415us) client_negotiation.cc:240] Negotiation successful
0806 02:45:48.956121 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":328,"threads_started":1}
I20260806 02:45:48.958792 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.66:0
--local_ip_for_outbound_sockets=127.1.210.66
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:39719
--builtin_ntp_servers=127.1.210.84:43271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 02:45:48.959476 6771 heartbeater.cc:347] Connected to a master server at 127.1.210.126:39719
I20260806 02:45:48.959827 6771 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:48.960597 6771 heartbeater.cc:511] Master 127.1.210.126:39719 requested a full tablet report, sending...
I20260806 02:45:48.963209 6585 ts_manager.cc:194] Registered new tserver with Master: 456d8c4f09124da19f1d4ecf9a319f82 (127.1.210.65:45265)
W20260806 02:45:49.306056 6775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:49.306402 6775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:49.306574 6775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:49.314045 6775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:49.314203 6775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:49.314386 6775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.66
I20260806 02:45:49.323881 6775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:43271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:39719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.66
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:49.325258 6775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:49.327106 6775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:49.340744 6783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:49.340770 6781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:49.341265 6780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:49.341864 6775 server_base.cc:1034] running on GCE node
I20260806 02:45:49.342479 6775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:49.343510 6775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:49.344801 6775 hybrid_clock.cc:648] HybridClock initialized: now 1785984349344726 us; error 47 us; skew 500 ppm
I20260806 02:45:49.345175 6775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:45:49.347734 6775 webserver.cc:460] Webserver started at http://127.1.210.66:41489/ using document root <none> and password file <none>
I20260806 02:45:49.348454 6775 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:49.348845 6775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:49.349256 6775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:49.352339 6775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-1/data/instance:
uuid: "9c03e3231bcf44ee80d273ffd6cc8928"
format_stamp: "Formatted at 2026-08-06 02:45:49 on dist-test-slave-8tdl"
server_key: "48e1c5f40d3351ea5377facffd782ea7"
server_key_iv: "a5c35cf1154ad32075f766cb589fd144"
server_key_version: "encryptionkey@0"
I20260806 02:45:49.353171 6775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance:
uuid: "9c03e3231bcf44ee80d273ffd6cc8928"
format_stamp: "Formatted at 2026-08-06 02:45:49 on dist-test-slave-8tdl"
server_key: "48e1c5f40d3351ea5377facffd782ea7"
server_key_iv: "a5c35cf1154ad32075f766cb589fd144"
server_key_version: "encryptionkey@0"
I20260806 02:45:49.358863 6775 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260806 02:45:49.362774 6789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:49.364219 6775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:49.364415 6775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-1/wal
uuid: "9c03e3231bcf44ee80d273ffd6cc8928"
format_stamp: "Formatted at 2026-08-06 02:45:49 on dist-test-slave-8tdl"
server_key: "48e1c5f40d3351ea5377facffd782ea7"
server_key_iv: "a5c35cf1154ad32075f766cb589fd144"
server_key_version: "encryptionkey@0"
I20260806 02:45:49.364691 6775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:49.370759 6775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:49.371632 6775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:49.371869 6775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:49.372763 6775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:49.374136 6775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:49.374224 6775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:49.374307 6775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:49.374369 6775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:49.408178 6775 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.66:45961
I20260806 02:45:49.408208 6901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.66:45961 every 8 connection(s)
I20260806 02:45:49.410079 6775 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
I20260806 02:45:49.413292 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 6775
I20260806 02:45:49.413466 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance
I20260806 02:45:49.413980 1865 external_mini_cluster.cc:1428] Setting key 62cbefde27197bc0795dd0e5d752048d
I20260806 02:45:49.417829 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.67:0
--local_ip_for_outbound_sockets=127.1.210.67
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:39719
--builtin_ntp_servers=127.1.210.84:43271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 02:45:49.420168 6773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.413059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:34801 (local address 127.1.210.126:39719)
0806 02:45:49.414746 (+ 1687us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:49.414753 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:49.415146 (+ 393us) server_negotiation.cc:407] Connection header received
0806 02:45:49.417559 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:49.417564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:49.417637 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:49.417759 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:49.419381 (+ 1622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:49.419387 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:49.419392 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:49.419414 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:49.419457 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:49.419470 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:49.419600 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:49.419795 (+ 195us) server_negotiation.cc:299] Negotiation successful
0806 02:45:49.419868 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1518}
I20260806 02:45:49.421211 6904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.413718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:39719 (local address 127.1.210.66:34801)
0806 02:45:49.414964 (+ 1246us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:49.415013 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:49.417245 (+ 2232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:49.418091 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:49.418102 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:49.418214 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:49.419049 (+ 835us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:45:49.419052 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:49.419097 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:45:49.419195 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:49.419568 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:49.419580 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:49.419588 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 02:45:49.419887 (+ 299us) client_negotiation.cc:240] Negotiation successful
0806 02:45:49.420186 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":434,"threads_started":1}
I20260806 02:45:49.422938 6902 heartbeater.cc:347] Connected to a master server at 127.1.210.126:39719
I20260806 02:45:49.423280 6902 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:49.423936 6902 heartbeater.cc:511] Master 127.1.210.126:39719 requested a full tablet report, sending...
I20260806 02:45:49.425707 6585 ts_manager.cc:194] Registered new tserver with Master: 9c03e3231bcf44ee80d273ffd6cc8928 (127.1.210.66:45961)
W20260806 02:45:49.762554 6905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:49.762974 6905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:49.763132 6905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:49.770998 6905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:49.771142 6905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:49.771322 6905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.67
I20260806 02:45:49.782289 6905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:43271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:39719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.67
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:49.783725 6905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:49.785665 6905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:49.800071 6910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:49.800195 6911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:49.800087 6913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:49.802207 6905 server_base.cc:1034] running on GCE node
I20260806 02:45:49.803035 6905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:49.804204 6905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:49.805449 6905 hybrid_clock.cc:648] HybridClock initialized: now 1785984349805393 us; error 47 us; skew 500 ppm
I20260806 02:45:49.805856 6905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:45:49.808842 6905 webserver.cc:460] Webserver started at http://127.1.210.67:41955/ using document root <none> and password file <none>
I20260806 02:45:49.809820 6905 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:49.809983 6905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:49.810346 6905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:49.813974 6905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-2/data/instance:
uuid: "52249f3ad5204240b5b4f0b58f9debff"
format_stamp: "Formatted at 2026-08-06 02:45:49 on dist-test-slave-8tdl"
server_key: "d68cc24c4bd7ef6350d20d7409b845ce"
server_key_iv: "eaef2f4e311f2e2af4abbf1921e0bffc"
server_key_version: "encryptionkey@0"
I20260806 02:45:49.815178 6905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance:
uuid: "52249f3ad5204240b5b4f0b58f9debff"
format_stamp: "Formatted at 2026-08-06 02:45:49 on dist-test-slave-8tdl"
server_key: "d68cc24c4bd7ef6350d20d7409b845ce"
server_key_iv: "eaef2f4e311f2e2af4abbf1921e0bffc"
server_key_version: "encryptionkey@0"
I20260806 02:45:49.821525 6905 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 02:45:49.825634 6919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:49.827484 6905 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260806 02:45:49.827698 6905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-2/wal
uuid: "52249f3ad5204240b5b4f0b58f9debff"
format_stamp: "Formatted at 2026-08-06 02:45:49 on dist-test-slave-8tdl"
server_key: "d68cc24c4bd7ef6350d20d7409b845ce"
server_key_iv: "eaef2f4e311f2e2af4abbf1921e0bffc"
server_key_version: "encryptionkey@0"
I20260806 02:45:49.827988 6905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:49.835296 6905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:49.836251 6905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:49.836546 6905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:49.837572 6905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:49.839174 6905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:49.839296 6905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:49.839427 6905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:49.839512 6905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:49.876988 6905 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.67:44167
I20260806 02:45:49.877079 7032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.67:44167 every 8 connection(s)
I20260806 02:45:49.878355 6905 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
I20260806 02:45:49.882972 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 6905
I20260806 02:45:49.883167 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance
I20260806 02:45:49.883634 1865 external_mini_cluster.cc:1428] Setting key fca6e86661fdc5497af8275e23926fe4
I20260806 02:45:49.887849 6773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.881131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:49341 (local address 127.1.210.126:39719)
0806 02:45:49.881322 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:49.881326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:49.882886 (+ 1560us) server_negotiation.cc:407] Connection header received
0806 02:45:49.885205 (+ 2319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:49.885211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:49.885284 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:49.885408 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:49.887174 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:49.887178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:49.887182 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:49.887199 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:49.887233 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:49.887246 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:49.887340 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:49.887579 (+ 239us) server_negotiation.cc:299] Negotiation successful
0806 02:45:49.887632 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 02:45:49.889118 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.881687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:39719 (local address 127.1.210.67:49341)
0806 02:45:49.882651 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:49.882771 (+ 120us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:49.884841 (+ 2070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:49.885753 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:49.885762 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:49.885993 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:49.886809 (+ 816us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:45:49.886812 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:49.886866 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:45:49.887001 (+ 135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:49.887362 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:49.887370 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:49.887376 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 02:45:49.887640 (+ 264us) client_negotiation.cc:240] Negotiation successful
0806 02:45:49.887853 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":282,"threads_started":1}
I20260806 02:45:49.890769 7033 heartbeater.cc:347] Connected to a master server at 127.1.210.126:39719
I20260806 02:45:49.891148 7033 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:49.891767 7033 heartbeater.cc:511] Master 127.1.210.126:39719 requested a full tablet report, sending...
I20260806 02:45:49.893244 6585 ts_manager.cc:194] Registered new tserver with Master: 52249f3ad5204240b5b4f0b58f9debff (127.1.210.67:44167)
I20260806 02:45:49.899518 1865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:45:49.907162 6773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.904509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33316 (local address 127.1.210.126:39719)
0806 02:45:49.904682 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:49.904687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:49.905170 (+ 483us) server_negotiation.cc:407] Connection header received
0806 02:45:49.905368 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:49.905372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:49.905425 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:49.905524 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:49.906475 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:49.906481 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:49.906486 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:49.906510 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:49.906543 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:49.906555 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:49.906710 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:49.906827 (+ 117us) server_negotiation.cc:299] Negotiation successful
0806 02:45:49.906892 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 02:45:49.911273 6585 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:45:49.918375 6585 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:45:49.944937 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.932499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:44167 (local address 127.0.0.1:54984)
0806 02:45:49.939375 (+ 6876us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:49.939418 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:49.939713 (+ 295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:49.941756 (+ 2043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:49.941765 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:49.941873 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:49.943456 (+ 1583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:45:49.943462 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:49.943504 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:45:49.943619 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:49.944201 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:49.944212 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:49.944220 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 02:45:49.944401 (+ 181us) client_negotiation.cc:240] Negotiation successful
0806 02:45:49.944494 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6618,"thread_start_us":6391,"threads_started":1}
I20260806 02:45:49.945251 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.933806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42600 (local address 127.1.210.66:45961)
0806 02:45:49.939326 (+ 5520us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:49.939340 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:49.941413 (+ 2073us) server_negotiation.cc:407] Connection header received
0806 02:45:49.941514 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:49.941525 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:49.941778 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:49.941930 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:49.944155 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:49.944163 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:49.944173 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:49.944192 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:49.944245 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:49.944264 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:49.944414 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:49.944786 (+ 372us) server_negotiation.cc:299] Negotiation successful
0806 02:45:49.944918 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5323,"thread_start_us":5198,"threads_started":1}
I20260806 02:45:49.946319 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.933113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:45961 (local address 127.0.0.1:42600)
0806 02:45:49.939866 (+ 6753us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:49.939888 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:49.940009 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:49.942843 (+ 2834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:49.942848 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:49.942880 (+ 32us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:49.943884 (+ 1004us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:45:49.943888 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:49.943905 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:45:49.943956 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:49.944605 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:49.944610 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:49.944614 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 02:45:49.945803 (+ 1189us) client_negotiation.cc:240] Negotiation successful
0806 02:45:49.945865 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6591,"thread_start_us":6425,"threads_started":1}
I20260806 02:45:49.946379 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.932746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54984 (local address 127.1.210.67:44167)
0806 02:45:49.935205 (+ 2459us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:49.935220 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:49.939559 (+ 4339us) server_negotiation.cc:407] Connection header received
0806 02:45:49.939889 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:49.939901 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:49.940199 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:49.940384 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:49.943828 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:49.943836 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:49.943848 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:49.943865 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:49.943917 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:49.943932 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:49.944139 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:49.944432 (+ 293us) server_negotiation.cc:299] Negotiation successful
0806 02:45:49.945927 (+ 1495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2219,"thread_start_us":1469,"threads_started":1}
I20260806 02:45:49.949302 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.940149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:45265 (local address 127.0.0.1:45008)
0806 02:45:49.943884 (+ 3735us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:49.943914 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:49.944025 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:49.947257 (+ 3232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:49.947262 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:49.947289 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:49.948036 (+ 747us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:45:49.948040 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:49.948058 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:45:49.948108 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:49.948656 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:49.948661 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:49.948667 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 02:45:49.948801 (+ 134us) client_negotiation.cc:240] Negotiation successful
0806 02:45:49.948857 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3568,"thread_start_us":3404,"threads_started":1}
I20260806 02:45:49.950006 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.940163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45008 (local address 127.1.210.65:45265)
0806 02:45:49.943375 (+ 3212us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:49.943385 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:49.944026 (+ 641us) server_negotiation.cc:407] Connection header received
0806 02:45:49.945748 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:49.945755 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:49.945993 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:49.946145 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:49.948282 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:49.948291 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:49.948305 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:49.948333 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:49.948399 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:49.948421 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:49.948597 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:49.949528 (+ 931us) server_negotiation.cc:299] Negotiation successful
0806 02:45:49.949641 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3008,"thread_start_us":2863,"threads_started":1}
I20260806 02:45:49.951102 6837 tablet_service.cc:1467] Processing CreateTablet for tablet 6df1d0c4b25b4a5dbe18bdff2d50104a (DEFAULT_TABLE table=test-table [id=3099fbead913409ea0fb75d615dee09a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:49.950793 6968 tablet_service.cc:1467] Processing CreateTablet for tablet 6df1d0c4b25b4a5dbe18bdff2d50104a (DEFAULT_TABLE table=test-table [id=3099fbead913409ea0fb75d615dee09a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:49.952715 6837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6df1d0c4b25b4a5dbe18bdff2d50104a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:49.953687 6968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6df1d0c4b25b4a5dbe18bdff2d50104a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:49.953958 6706 tablet_service.cc:1467] Processing CreateTablet for tablet 6df1d0c4b25b4a5dbe18bdff2d50104a (DEFAULT_TABLE table=test-table [id=3099fbead913409ea0fb75d615dee09a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:49.955102 6706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6df1d0c4b25b4a5dbe18bdff2d50104a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:49.966835 6771 heartbeater.cc:503] Master 127.1.210.126:39719 was elected leader, sending a full tablet report...
I20260806 02:45:49.968931 7051 tablet_bootstrap.cc:492] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 456d8c4f09124da19f1d4ecf9a319f82: Bootstrap starting.
I20260806 02:45:49.972635 7053 tablet_bootstrap.cc:492] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 52249f3ad5204240b5b4f0b58f9debff: Bootstrap starting.
I20260806 02:45:49.972734 7051 tablet_bootstrap.cc:654] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 456d8c4f09124da19f1d4ecf9a319f82: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:49.974112 7051 log.cc:824] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 456d8c4f09124da19f1d4ecf9a319f82: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:49.975565 7052 tablet_bootstrap.cc:492] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 9c03e3231bcf44ee80d273ffd6cc8928: Bootstrap starting.
I20260806 02:45:49.978155 7053 tablet_bootstrap.cc:654] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 52249f3ad5204240b5b4f0b58f9debff: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:49.978514 7052 tablet_bootstrap.cc:654] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 9c03e3231bcf44ee80d273ffd6cc8928: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:49.979617 7052 log.cc:824] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 9c03e3231bcf44ee80d273ffd6cc8928: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:49.979791 7053 log.cc:824] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 52249f3ad5204240b5b4f0b58f9debff: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:49.982813 7052 tablet_bootstrap.cc:492] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 9c03e3231bcf44ee80d273ffd6cc8928: No bootstrap required, opened a new log
I20260806 02:45:49.982867 7051 tablet_bootstrap.cc:492] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 456d8c4f09124da19f1d4ecf9a319f82: No bootstrap required, opened a new log
I20260806 02:45:49.983129 7051 ts_tablet_manager.cc:1397] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 456d8c4f09124da19f1d4ecf9a319f82: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.012s
I20260806 02:45:49.983150 7052 ts_tablet_manager.cc:1397] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 9c03e3231bcf44ee80d273ffd6cc8928: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260806 02:45:49.985209 7053 tablet_bootstrap.cc:492] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 52249f3ad5204240b5b4f0b58f9debff: No bootstrap required, opened a new log
I20260806 02:45:49.985849 7053 ts_tablet_manager.cc:1397] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 52249f3ad5204240b5b4f0b58f9debff: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260806 02:45:49.990651 7051 raft_consensus.cc:348] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 456d8c4f09124da19f1d4ecf9a319f82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52249f3ad5204240b5b4f0b58f9debff" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44167 } } peers { permanent_uuid: "456d8c4f09124da19f1d4ecf9a319f82" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 45265 } } peers { permanent_uuid: "9c03e3231bcf44ee80d273ffd6cc8928" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45961 } }
I20260806 02:45:49.991144 7051 raft_consensus.cc:374] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 456d8c4f09124da19f1d4ecf9a319f82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:49.991266 7051 raft_consensus.cc:729] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 456d8c4f09124da19f1d4ecf9a319f82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 456d8c4f09124da19f1d4ecf9a319f82, State: Initialized, Role: FOLLOWER
I20260806 02:45:49.991922 7051 consensus_queue.cc:260] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 456d8c4f09124da19f1d4ecf9a319f82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52249f3ad5204240b5b4f0b58f9debff" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44167 } } peers { permanent_uuid: "456d8c4f09124da19f1d4ecf9a319f82" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 45265 } } peers { permanent_uuid: "9c03e3231bcf44ee80d273ffd6cc8928" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45961 } }
I20260806 02:45:49.993798 7051 ts_tablet_manager.cc:1428] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 456d8c4f09124da19f1d4ecf9a319f82: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260806 02:45:49.996093 7052 raft_consensus.cc:348] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 9c03e3231bcf44ee80d273ffd6cc8928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52249f3ad5204240b5b4f0b58f9debff" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44167 } } peers { permanent_uuid: "456d8c4f09124da19f1d4ecf9a319f82" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 45265 } } peers { permanent_uuid: "9c03e3231bcf44ee80d273ffd6cc8928" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45961 } }
I20260806 02:45:49.996616 7052 raft_consensus.cc:374] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 9c03e3231bcf44ee80d273ffd6cc8928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:49.996731 7052 raft_consensus.cc:729] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 9c03e3231bcf44ee80d273ffd6cc8928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c03e3231bcf44ee80d273ffd6cc8928, State: Initialized, Role: FOLLOWER
I20260806 02:45:49.997465 7052 consensus_queue.cc:260] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 9c03e3231bcf44ee80d273ffd6cc8928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52249f3ad5204240b5b4f0b58f9debff" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44167 } } peers { permanent_uuid: "456d8c4f09124da19f1d4ecf9a319f82" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 45265 } } peers { permanent_uuid: "9c03e3231bcf44ee80d273ffd6cc8928" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45961 } }
I20260806 02:45:49.997925 7053 raft_consensus.cc:348] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 52249f3ad5204240b5b4f0b58f9debff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52249f3ad5204240b5b4f0b58f9debff" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44167 } } peers { permanent_uuid: "456d8c4f09124da19f1d4ecf9a319f82" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 45265 } } peers { permanent_uuid: "9c03e3231bcf44ee80d273ffd6cc8928" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45961 } }
I20260806 02:45:49.998469 7053 raft_consensus.cc:374] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 52249f3ad5204240b5b4f0b58f9debff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:49.998638 7053 raft_consensus.cc:729] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 52249f3ad5204240b5b4f0b58f9debff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52249f3ad5204240b5b4f0b58f9debff, State: Initialized, Role: FOLLOWER
I20260806 02:45:49.998762 6902 heartbeater.cc:503] Master 127.1.210.126:39719 was elected leader, sending a full tablet report...
I20260806 02:45:49.999598 7052 ts_tablet_manager.cc:1428] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 9c03e3231bcf44ee80d273ffd6cc8928: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260806 02:45:49.999449 7053 consensus_queue.cc:260] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 52249f3ad5204240b5b4f0b58f9debff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52249f3ad5204240b5b4f0b58f9debff" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44167 } } peers { permanent_uuid: "456d8c4f09124da19f1d4ecf9a319f82" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 45265 } } peers { permanent_uuid: "9c03e3231bcf44ee80d273ffd6cc8928" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45961 } }
I20260806 02:45:50.000665 7033 heartbeater.cc:503] Master 127.1.210.126:39719 was elected leader, sending a full tablet report...
I20260806 02:45:50.001690 7053 ts_tablet_manager.cc:1428] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 52249f3ad5204240b5b4f0b58f9debff: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260806 02:45:50.045872 7058 raft_consensus.cc:482] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 9c03e3231bcf44ee80d273ffd6cc8928 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:45:50.046216 7058 raft_consensus.cc:504] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 9c03e3231bcf44ee80d273ffd6cc8928 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52249f3ad5204240b5b4f0b58f9debff" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44167 } } peers { permanent_uuid: "456d8c4f09124da19f1d4ecf9a319f82" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 45265 } } peers { permanent_uuid: "9c03e3231bcf44ee80d273ffd6cc8928" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45961 } }
I20260806 02:45:50.048151 7058 leader_election.cc:288] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 9c03e3231bcf44ee80d273ffd6cc8928 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52249f3ad5204240b5b4f0b58f9debff (127.1.210.67:44167), 456d8c4f09124da19f1d4ecf9a319f82 (127.1.210.65:45265)
I20260806 02:45:50.052094 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.048632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:54025 (local address 127.1.210.67:44167)
0806 02:45:50.048853 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:50.048862 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:50.049524 (+ 662us) server_negotiation.cc:407] Connection header received
0806 02:45:50.049755 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:50.049761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:50.049833 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:50.049912 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:50.051413 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:50.051420 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:50.051425 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:50.051448 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:50.051481 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:50.051493 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:50.051681 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:50.051729 (+ 48us) server_negotiation.cc:299] Negotiation successful
0806 02:45:50.051798 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:45:50.052273 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.049712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:35233 (local address 127.1.210.65:45265)
0806 02:45:50.049894 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:50.049899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:50.050208 (+ 309us) server_negotiation.cc:407] Connection header received
0806 02:45:50.050302 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:50.050310 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:50.050378 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:50.050496 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:50.051620 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:50.051626 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:50.051630 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:50.051650 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:50.051691 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:50.051709 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:50.051841 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:50.051950 (+ 109us) server_negotiation.cc:299] Negotiation successful
0806 02:45:50.052012 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 02:45:50.052413 7060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.048553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:44167 (local address 127.1.210.66:54025)
0806 02:45:50.049359 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:50.049376 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:50.049560 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:50.050130 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:50.050133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:50.050179 (+ 46us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:50.051182 (+ 1003us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:45:50.051185 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:50.051200 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:45:50.051241 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:50.051630 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:50.051633 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:50.051636 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 02:45:50.051742 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 02:45:50.051818 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":484,"threads_started":1}
I20260806 02:45:50.052623 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.049457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:45265 (local address 127.1.210.66:35233)
0806 02:45:50.050089 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:50.050104 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:50.050196 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:50.050588 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:50.050594 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:50.050622 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:50.051414 (+ 792us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:45:50.051416 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:50.051429 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:45:50.051465 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:50.051829 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:50.051834 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:50.051839 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 02:45:50.052064 (+ 225us) client_negotiation.cc:240] Negotiation successful
0806 02:45:50.052126 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":311,"threads_started":1}
I20260806 02:45:50.053563 6988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6df1d0c4b25b4a5dbe18bdff2d50104a" candidate_uuid: "9c03e3231bcf44ee80d273ffd6cc8928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52249f3ad5204240b5b4f0b58f9debff" is_pre_election: true
I20260806 02:45:50.053566 6726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6df1d0c4b25b4a5dbe18bdff2d50104a" candidate_uuid: "9c03e3231bcf44ee80d273ffd6cc8928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "456d8c4f09124da19f1d4ecf9a319f82" is_pre_election: true
I20260806 02:45:50.054075 6726 raft_consensus.cc:2454] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 456d8c4f09124da19f1d4ecf9a319f82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c03e3231bcf44ee80d273ffd6cc8928 in term 0.
I20260806 02:45:50.054078 6988 raft_consensus.cc:2454] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 52249f3ad5204240b5b4f0b58f9debff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c03e3231bcf44ee80d273ffd6cc8928 in term 0.
I20260806 02:45:50.054788 6791 leader_election.cc:302] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 9c03e3231bcf44ee80d273ffd6cc8928 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 456d8c4f09124da19f1d4ecf9a319f82, 9c03e3231bcf44ee80d273ffd6cc8928; no voters:
I20260806 02:45:50.055234 7058 raft_consensus.cc:2781] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 9c03e3231bcf44ee80d273ffd6cc8928 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:45:50.055418 7058 raft_consensus.cc:482] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 9c03e3231bcf44ee80d273ffd6cc8928 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:45:50.055579 7058 raft_consensus.cc:3037] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 9c03e3231bcf44ee80d273ffd6cc8928 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:50.058154 7058 raft_consensus.cc:504] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 9c03e3231bcf44ee80d273ffd6cc8928 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52249f3ad5204240b5b4f0b58f9debff" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44167 } } peers { permanent_uuid: "456d8c4f09124da19f1d4ecf9a319f82" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 45265 } } peers { permanent_uuid: "9c03e3231bcf44ee80d273ffd6cc8928" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45961 } }
I20260806 02:45:50.058915 7058 leader_election.cc:288] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 9c03e3231bcf44ee80d273ffd6cc8928 [CANDIDATE]: Term 1 election: Requested vote from peers 52249f3ad5204240b5b4f0b58f9debff (127.1.210.67:44167), 456d8c4f09124da19f1d4ecf9a319f82 (127.1.210.65:45265)
I20260806 02:45:50.059540 6988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6df1d0c4b25b4a5dbe18bdff2d50104a" candidate_uuid: "9c03e3231bcf44ee80d273ffd6cc8928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52249f3ad5204240b5b4f0b58f9debff"
I20260806 02:45:50.059587 6726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6df1d0c4b25b4a5dbe18bdff2d50104a" candidate_uuid: "9c03e3231bcf44ee80d273ffd6cc8928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "456d8c4f09124da19f1d4ecf9a319f82"
I20260806 02:45:50.059798 6988 raft_consensus.cc:3037] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 52249f3ad5204240b5b4f0b58f9debff [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:50.059798 6726 raft_consensus.cc:3037] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 456d8c4f09124da19f1d4ecf9a319f82 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:50.063338 6726 raft_consensus.cc:2454] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 456d8c4f09124da19f1d4ecf9a319f82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c03e3231bcf44ee80d273ffd6cc8928 in term 1.
I20260806 02:45:50.063338 6988 raft_consensus.cc:2454] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 52249f3ad5204240b5b4f0b58f9debff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c03e3231bcf44ee80d273ffd6cc8928 in term 1.
I20260806 02:45:50.063827 6791 leader_election.cc:302] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 9c03e3231bcf44ee80d273ffd6cc8928 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 456d8c4f09124da19f1d4ecf9a319f82, 9c03e3231bcf44ee80d273ffd6cc8928; no voters:
I20260806 02:45:50.064217 7058 raft_consensus.cc:2781] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 9c03e3231bcf44ee80d273ffd6cc8928 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:50.064792 7058 raft_consensus.cc:686] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 9c03e3231bcf44ee80d273ffd6cc8928 [term 1 LEADER]: Becoming Leader. State: Replica: 9c03e3231bcf44ee80d273ffd6cc8928, State: Running, Role: LEADER
I20260806 02:45:50.065340 7058 consensus_queue.cc:237] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 9c03e3231bcf44ee80d273ffd6cc8928 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52249f3ad5204240b5b4f0b58f9debff" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44167 } } peers { permanent_uuid: "456d8c4f09124da19f1d4ecf9a319f82" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 45265 } } peers { permanent_uuid: "9c03e3231bcf44ee80d273ffd6cc8928" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45961 } }
I20260806 02:45:50.070463 6584 catalog_manager.cc:5495] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 9c03e3231bcf44ee80d273ffd6cc8928 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c03e3231bcf44ee80d273ffd6cc8928 (127.1.210.66). New cstate: current_term: 1 leader_uuid: "9c03e3231bcf44ee80d273ffd6cc8928" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52249f3ad5204240b5b4f0b58f9debff" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "456d8c4f09124da19f1d4ecf9a319f82" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 45265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c03e3231bcf44ee80d273ffd6cc8928" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45961 } health_report { overall_health: HEALTHY } } }
I20260806 02:45:50.102718 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.100068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42604 (local address 127.1.210.66:45961)
0806 02:45:50.100283 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:50.100290 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:50.100396 (+ 106us) server_negotiation.cc:407] Connection header received
0806 02:45:50.100676 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:50.100681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:50.100766 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:50.100914 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:50.101978 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:50.101984 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:50.101988 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:50.102007 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:50.102069 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:50.102085 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:50.102215 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:50.102313 (+ 98us) server_negotiation.cc:299] Negotiation successful
0806 02:45:50.102380 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 02:45:50.111413 6726 raft_consensus.cc:1261] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 456d8c4f09124da19f1d4ecf9a319f82 [term 1 FOLLOWER]: Refusing update from remote peer 9c03e3231bcf44ee80d273ffd6cc8928: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:50.111413 6988 raft_consensus.cc:1261] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 52249f3ad5204240b5b4f0b58f9debff [term 1 FOLLOWER]: Refusing update from remote peer 9c03e3231bcf44ee80d273ffd6cc8928: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:50.112669 7062 consensus_queue.cc:1035] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 9c03e3231bcf44ee80d273ffd6cc8928 [LEADER]: Connected to new peer: Peer: permanent_uuid: "456d8c4f09124da19f1d4ecf9a319f82" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 45265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:50.113124 7058 consensus_queue.cc:1035] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 9c03e3231bcf44ee80d273ffd6cc8928 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52249f3ad5204240b5b4f0b58f9debff" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:50.140156 6584 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:50.140496 6584 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:50.144858 6584 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a1fea205d6b4a69a630fd5f9680e883: Sending DeleteTablet for 3 replicas of tablet 6df1d0c4b25b4a5dbe18bdff2d50104a
I20260806 02:45:50.146430 6968 tablet_service.cc:1514] Processing DeleteTablet for tablet 6df1d0c4b25b4a5dbe18bdff2d50104a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:50 UTC) from {username='slave'} at 127.0.0.1:54984
I20260806 02:45:50.146395 6837 tablet_service.cc:1514] Processing DeleteTablet for tablet 6df1d0c4b25b4a5dbe18bdff2d50104a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:50 UTC) from {username='slave'} at 127.0.0.1:42600
I20260806 02:45:50.146626 6706 tablet_service.cc:1514] Processing DeleteTablet for tablet 6df1d0c4b25b4a5dbe18bdff2d50104a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:50 UTC) from {username='slave'} at 127.0.0.1:45008
I20260806 02:45:50.148893 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 6644
I20260806 02:45:50.159624 7075 tablet_replica.cc:330] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 9c03e3231bcf44ee80d273ffd6cc8928: stopping tablet replica
I20260806 02:45:50.159755 7074 tablet_replica.cc:330] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 52249f3ad5204240b5b4f0b58f9debff: stopping tablet replica
I20260806 02:45:50.160434 7074 raft_consensus.cc:2229] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 52249f3ad5204240b5b4f0b58f9debff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:50.160434 7075 raft_consensus.cc:2229] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 9c03e3231bcf44ee80d273ffd6cc8928 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:45:50.161113 7074 raft_consensus.cc:2258] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 52249f3ad5204240b5b4f0b58f9debff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:50.161155 7075 raft_consensus.cc:2258] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 9c03e3231bcf44ee80d273ffd6cc8928 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:50.162350 7076 tablet_replica.cc:330] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 456d8c4f09124da19f1d4ecf9a319f82: stopping tablet replica
I20260806 02:45:50.162891 7076 raft_consensus.cc:2229] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 456d8c4f09124da19f1d4ecf9a319f82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:50.163777 7076 raft_consensus.cc:2258] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 456d8c4f09124da19f1d4ecf9a319f82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:50.163954 7075 ts_tablet_manager.cc:1905] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 9c03e3231bcf44ee80d273ffd6cc8928: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:50.163960 7074 ts_tablet_manager.cc:1905] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 52249f3ad5204240b5b4f0b58f9debff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:50.165380 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.150900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45022 (local address 127.1.210.65:45265)
0806 02:45:50.151149 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:50.151158 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:50.162078 (+ 10920us) server_negotiation.cc:407] Connection header received
0806 02:45:50.162416 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:50.162423 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:50.162537 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:50.162619 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:50.164559 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:50.164565 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:50.164570 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:50.164594 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:50.164638 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:50.164655 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:50.164763 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:50.165017 (+ 254us) server_negotiation.cc:299] Negotiation successful
0806 02:45:50.165087 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 02:45:50.166878 6766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:45:50.172680 7074 ts_tablet_manager.cc:1918] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 52249f3ad5204240b5b4f0b58f9debff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:45:50.172951 7074 log.cc:1196] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 52249f3ad5204240b5b4f0b58f9debff: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-2/wal/wals/6df1d0c4b25b4a5dbe18bdff2d50104a
I20260806 02:45:50.173610 7074 ts_tablet_manager.cc:1939] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 52249f3ad5204240b5b4f0b58f9debff: Deleting consensus metadata
I20260806 02:45:50.173744 7075 ts_tablet_manager.cc:1918] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 9c03e3231bcf44ee80d273ffd6cc8928: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:45:50.174022 7075 log.cc:1196] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 9c03e3231bcf44ee80d273ffd6cc8928: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984304136196-1865-0/minicluster-data/ts-1/wal/wals/6df1d0c4b25b4a5dbe18bdff2d50104a
I20260806 02:45:50.174820 7075 ts_tablet_manager.cc:1939] T 6df1d0c4b25b4a5dbe18bdff2d50104a P 9c03e3231bcf44ee80d273ffd6cc8928: Deleting consensus metadata
I20260806 02:45:50.176030 6570 catalog_manager.cc:4841] TS 52249f3ad5204240b5b4f0b58f9debff (127.1.210.67:44167): tablet 6df1d0c4b25b4a5dbe18bdff2d50104a (table test-table [id=3099fbead913409ea0fb75d615dee09a]) successfully deleted
I20260806 02:45:50.177639 6572 catalog_manager.cc:4841] TS 9c03e3231bcf44ee80d273ffd6cc8928 (127.1.210.66:45961): tablet 6df1d0c4b25b4a5dbe18bdff2d50104a (table test-table [id=3099fbead913409ea0fb75d615dee09a]) successfully deleted
I20260806 02:45:50.265806 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 6644
I20260806 02:45:50.284461 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 6775
W20260806 02:45:50.284490 6570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.210.65:45265 (error 108)
I20260806 02:45:50.286598 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.286107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:45265 (local address 127.0.0.1:45034)
0806 02:45:50.286286 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:50.286425 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:45265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260806 02:45:50.286876 6570 catalog_manager.cc:4584] TS 456d8c4f09124da19f1d4ecf9a319f82 (127.1.210.65:45265): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6df1d0c4b25b4a5dbe18bdff2d50104a: Network error: Client connection negotiation failed: client connection to 127.1.210.65:45265: connect: Connection refused (error 111)
I20260806 02:45:50.288441 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.285374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42618 (local address 127.1.210.66:45961)
0806 02:45:50.285600 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:50.285609 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:50.286287 (+ 678us) server_negotiation.cc:407] Connection header received
0806 02:45:50.286388 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:50.286395 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:50.286474 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:50.286580 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:50.287683 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:50.287688 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:50.287692 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:50.287714 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:50.287748 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:50.287760 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:50.287884 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:50.288050 (+ 166us) server_negotiation.cc:299] Negotiation successful
0806 02:45:50.288113 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 02:45:50.289219 6897 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:45:50.340754 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.340181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:45265 (local address 127.0.0.1:45044)
0806 02:45:50.340423 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:50.340564 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:45265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260806 02:45:50.341997 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.341599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:45265 (local address 127.0.0.1:45046)
0806 02:45:50.341764 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:50.341865 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:45265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 02:45:50.403083 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.402464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:45265 (local address 127.0.0.1:45058)
0806 02:45:50.402714 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:50.402872 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:45265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 02:45:50.404096 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 6775
I20260806 02:45:50.404253 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.403803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:45265 (local address 127.0.0.1:45072)
0806 02:45:50.404009 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:50.404100 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:45265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260806 02:45:50.423132 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 6905
I20260806 02:45:50.426256 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.423995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54986 (local address 127.1.210.67:44167)
0806 02:45:50.424232 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:50.424240 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:50.424261 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:45:50.424461 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:50.424466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:50.424557 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:50.424665 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:50.425584 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:50.425590 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:50.425595 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:50.425613 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:50.425650 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:50.425662 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:50.425801 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:50.425881 (+ 80us) server_negotiation.cc:299] Negotiation successful
0806 02:45:50.425944 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260806 02:45:50.426900 7028 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:45:50.485224 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.484566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:45265 (local address 127.0.0.1:45082)
0806 02:45:50.484882 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:50.485016 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:45265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260806 02:45:50.486606 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.486067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:45265 (local address 127.0.0.1:45098)
0806 02:45:50.486322 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:50.486426 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:45265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260806 02:45:50.518139 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 6905
I20260806 02:45:50.536993 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 6554
I20260806 02:45:50.538408 6615 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:45:50.649749 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 6554
2026-08-06T02:45:50Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2764 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260806 02:45:50.694553 1865 mini_oidc.cc:152] Starting JWKS server
I20260806 02:45:50.697630 1865 webserver.cc:460] Webserver started at https://127.0.0.1:40397/ using document root <none> and password file <none>
I20260806 02:45:50.697973 1865 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 02:45:50.698630 1865 webserver.cc:460] Webserver started at http://127.0.0.1:34837/ using document root <none> and password file <none>
2026-08-06T02:45:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:50Z Disabled control of system clock
I20260806 02:45:50.732492 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:45827
--webserver_interface=127.1.210.126
--webserver_port=0
--builtin_ntp_servers=127.1.210.84:39395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:45827
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40397/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984304136196-1865-0/testchainca.cert with env {}
W20260806 02:45:51.086156 7088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:51.086549 7088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:51.086771 7088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:51.095587 7088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:51.095770 7088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:51.095858 7088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:51.095970 7088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:51.096056 7088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 02:45:51.096156 7088 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 02:45:51.096232 7088 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40397/jwks.json
I20260806 02:45:51.107448 7088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:39395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:45827
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:45827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40397/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984304136196-1865-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:51.108876 7088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:51.110267 7088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:51.124135 7093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:51.124213 7096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:51.124135 7094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:51.125090 7088 server_base.cc:1034] running on GCE node
I20260806 02:45:51.125957 7088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:51.127563 7088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:51.128821 7088 hybrid_clock.cc:648] HybridClock initialized: now 1785984351128768 us; error 42 us; skew 500 ppm
I20260806 02:45:51.129325 7088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:45:51.132261 7088 webserver.cc:460] Webserver started at http://127.1.210.126:40169/ using document root <none> and password file <none>
I20260806 02:45:51.133023 7088 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:51.133144 7088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:51.133473 7088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:51.136684 7088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "b04a932e3a0e4849b8e43ff208e5ca93"
format_stamp: "Formatted at 2026-08-06 02:45:51 on dist-test-slave-8tdl"
server_key: "f1a661155c32d48dced66049b688c69a"
server_key_iv: "a6fc0c44947fe85d38d4a31457c58a37"
server_key_version: "encryptionkey@0"
I20260806 02:45:51.137532 7088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "b04a932e3a0e4849b8e43ff208e5ca93"
format_stamp: "Formatted at 2026-08-06 02:45:51 on dist-test-slave-8tdl"
server_key: "f1a661155c32d48dced66049b688c69a"
server_key_iv: "a6fc0c44947fe85d38d4a31457c58a37"
server_key_version: "encryptionkey@0"
I20260806 02:45:51.143311 7088 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260806 02:45:51.147445 7102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:51.149066 7088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:45:51.149303 7088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "b04a932e3a0e4849b8e43ff208e5ca93"
format_stamp: "Formatted at 2026-08-06 02:45:51 on dist-test-slave-8tdl"
server_key: "f1a661155c32d48dced66049b688c69a"
server_key_iv: "a6fc0c44947fe85d38d4a31457c58a37"
server_key_version: "encryptionkey@0"
I20260806 02:45:51.149600 7088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:51.168331 7088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:51.169265 7088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:51.169602 7088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:51.190402 7088 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:45827
I20260806 02:45:51.190420 7153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:45827 every 8 connection(s)
I20260806 02:45:51.192140 7088 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:45:51.196578 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 7088
I20260806 02:45:51.196660 7154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:51.196826 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:45:51.197688 1865 external_mini_cluster.cc:1428] Setting key db8c4b3f7618fea7e4fc4a639ca2ecb0
I20260806 02:45:51.211925 7154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b04a932e3a0e4849b8e43ff208e5ca93: Bootstrap starting.
I20260806 02:45:51.214488 7157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:51.199869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59886 (local address 127.1.210.126:45827)
0806 02:45:51.201319 (+ 1450us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:51.201341 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:51.201411 (+ 70us) server_negotiation.cc:407] Connection header received
0806 02:45:51.203794 (+ 2383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:51.203825 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:51.204449 (+ 624us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:51.204879 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:51.205635 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:51.207144 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:51.209203 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:51.210002 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:51.210148 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:51.211650 (+ 1502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:51.211659 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:51.211676 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:51.212399 (+ 723us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:51.212500 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:51.212530 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:51.212631 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:51.212906 (+ 275us) server_negotiation.cc:299] Negotiation successful
0806 02:45:51.213397 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":438,"threads_started":1}
I20260806 02:45:51.215510 7154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b04a932e3a0e4849b8e43ff208e5ca93: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:51.216621 7154 log.cc:824] T 00000000000000000000000000000000 P b04a932e3a0e4849b8e43ff208e5ca93: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:51.218806 1865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:45:51.219668 7154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b04a932e3a0e4849b8e43ff208e5ca93: No bootstrap required, opened a new log
I20260806 02:45:51.227708 7154 raft_consensus.cc:348] T 00000000000000000000000000000000 P b04a932e3a0e4849b8e43ff208e5ca93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b04a932e3a0e4849b8e43ff208e5ca93" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45827 } }
I20260806 02:45:51.228252 7154 raft_consensus.cc:374] T 00000000000000000000000000000000 P b04a932e3a0e4849b8e43ff208e5ca93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:51.228415 7154 raft_consensus.cc:729] T 00000000000000000000000000000000 P b04a932e3a0e4849b8e43ff208e5ca93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b04a932e3a0e4849b8e43ff208e5ca93, State: Initialized, Role: FOLLOWER
I20260806 02:45:51.229144 7154 consensus_queue.cc:260] T 00000000000000000000000000000000 P b04a932e3a0e4849b8e43ff208e5ca93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b04a932e3a0e4849b8e43ff208e5ca93" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45827 } }
I20260806 02:45:51.229413 7154 raft_consensus.cc:388] T 00000000000000000000000000000000 P b04a932e3a0e4849b8e43ff208e5ca93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:45:51.229562 7154 raft_consensus.cc:482] T 00000000000000000000000000000000 P b04a932e3a0e4849b8e43ff208e5ca93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:45:51.229748 7154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b04a932e3a0e4849b8e43ff208e5ca93 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:51.232107 7154 raft_consensus.cc:504] T 00000000000000000000000000000000 P b04a932e3a0e4849b8e43ff208e5ca93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b04a932e3a0e4849b8e43ff208e5ca93" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45827 } }
I20260806 02:45:51.232662 7154 leader_election.cc:302] T 00000000000000000000000000000000 P b04a932e3a0e4849b8e43ff208e5ca93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b04a932e3a0e4849b8e43ff208e5ca93; no voters:
I20260806 02:45:51.233282 7154 leader_election.cc:288] T 00000000000000000000000000000000 P b04a932e3a0e4849b8e43ff208e5ca93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:45:51.233517 7159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b04a932e3a0e4849b8e43ff208e5ca93 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:51.234259 7159 raft_consensus.cc:686] T 00000000000000000000000000000000 P b04a932e3a0e4849b8e43ff208e5ca93 [term 1 LEADER]: Becoming Leader. State: Replica: b04a932e3a0e4849b8e43ff208e5ca93, State: Running, Role: LEADER
I20260806 02:45:51.234828 7159 consensus_queue.cc:237] T 00000000000000000000000000000000 P b04a932e3a0e4849b8e43ff208e5ca93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b04a932e3a0e4849b8e43ff208e5ca93" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45827 } }
I20260806 02:45:51.235922 7154 sys_catalog.cc:564] T 00000000000000000000000000000000 P b04a932e3a0e4849b8e43ff208e5ca93 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:51.238354 7160 sys_catalog.cc:455] T 00000000000000000000000000000000 P b04a932e3a0e4849b8e43ff208e5ca93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b04a932e3a0e4849b8e43ff208e5ca93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b04a932e3a0e4849b8e43ff208e5ca93" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45827 } } }
I20260806 02:45:51.238319 7161 sys_catalog.cc:455] T 00000000000000000000000000000000 P b04a932e3a0e4849b8e43ff208e5ca93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b04a932e3a0e4849b8e43ff208e5ca93. Latest consensus state: current_term: 1 leader_uuid: "b04a932e3a0e4849b8e43ff208e5ca93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b04a932e3a0e4849b8e43ff208e5ca93" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45827 } } }
I20260806 02:45:51.238874 7160 sys_catalog.cc:458] T 00000000000000000000000000000000 P b04a932e3a0e4849b8e43ff208e5ca93 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:51.238972 7161 sys_catalog.cc:458] T 00000000000000000000000000000000 P b04a932e3a0e4849b8e43ff208e5ca93 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:51.241211 7165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:51.250527 7165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:45:51.260881 7165 catalog_manager.cc:1347] Generated new cluster ID: 3dff6430bb3b4d7e9c879bc665ddb984
I20260806 02:45:51.261108 7165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:51.272272 7165 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:45:51.273754 7165 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:51.284722 7165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b04a932e3a0e4849b8e43ff208e5ca93: Generated new TSK 0
I20260806 02:45:51.285712 7165 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 02:45:51.332258 7184 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260806 02:45:51.332530 7184 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 02:45:51.335088 7157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:51.309431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59902 (local address 127.1.210.126:45827)
0806 02:45:51.309650 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:51.309657 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:51.310349 (+ 692us) server_negotiation.cc:407] Connection header received
0806 02:45:51.310772 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:51.310779 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:51.310904 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:51.311079 (+ 175us) server_negotiation.cc:226] Negotiated authn=JWT
0806 02:45:51.311675 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:51.312960 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:51.314642 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:51.315403 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:51.315554 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:51.316169 (+ 615us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 02:45:51.334245 (+ 18076us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40397/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 02:45:51.334691 (+ 446us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59902: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40397/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":72}
W20260806 02:45:51.335232 7157 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59902: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40397/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260806 02:45:51.337148 7183 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:51.309362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45827 (local address 127.0.0.1:59902)
0806 02:45:51.310223 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:51.310246 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:51.310429 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:51.311148 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:51.311157 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:51.311171 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0806 02:45:51.311501 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:51.311510 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:51.313142 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:51.313147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:51.314480 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:51.314494 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:51.315616 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:51.315622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:51.315771 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:51.315848 (+ 77us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 02:45:51.334656 (+ 18808us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40397/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 02:45:51.335121 (+ 465us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.210.126:45827: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40397/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":467,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.1.210.126:45827: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40397/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260806 02:45:51.339341 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 7088
I20260806 02:45:51.340577 7149 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:45:51.438853 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 7088
2026-08-06T02:45:51Z chronyd exiting
I20260806 02:45:51.491088 1865 test_util.cc:175] -----------------------------------------------
I20260806 02:45:51.491250 1865 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984304136196-1865-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (801 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260806 02:45:51.495255 1865 mini_oidc.cc:152] Starting JWKS server
I20260806 02:45:51.497223 1865 webserver.cc:460] Webserver started at https://127.0.0.1:37461/ using document root <none> and password file <none>
I20260806 02:45:51.497450 1865 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 02:45:51.498198 1865 webserver.cc:460] Webserver started at http://127.0.0.1:37207/ using document root <none> and password file <none>
2026-08-06T02:45:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:51Z Disabled control of system clock
I20260806 02:45:51.532392 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:39071
--webserver_interface=127.1.210.126
--webserver_port=0
--builtin_ntp_servers=127.1.210.84:45101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:39071
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37461/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984304136196-1865-0/testchainca.cert with env {}
W20260806 02:45:51.884542 7191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:51.884948 7191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:51.885113 7191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:51.892699 7191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:51.892879 7191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:51.893000 7191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:51.893114 7191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:51.893198 7191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 02:45:51.893292 7191 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 02:45:51.893381 7191 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37461/jwks.json
I20260806 02:45:51.903950 7191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:45101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:39071
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:39071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37461/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984304136196-1865-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:51.905383 7191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:51.906859 7191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:51.920611 7199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:51.920779 7196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:51.920574 7197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:51.921773 7191 server_base.cc:1034] running on GCE node
I20260806 02:45:51.923089 7191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:51.925539 7191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:51.926903 7191 hybrid_clock.cc:648] HybridClock initialized: now 1785984351926778 us; error 119 us; skew 500 ppm
I20260806 02:45:51.927392 7191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:45:51.930368 7191 webserver.cc:460] Webserver started at http://127.1.210.126:36281/ using document root <none> and password file <none>
I20260806 02:45:51.931154 7191 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:51.931252 7191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:51.931531 7191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:51.934494 7191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "65f62e6fe4e244f0938a30a6ba603591"
format_stamp: "Formatted at 2026-08-06 02:45:51 on dist-test-slave-8tdl"
server_key: "4e18da3ba30446513872c4d069bce1e7"
server_key_iv: "c5742854028fa9a911540af3ba878ff1"
server_key_version: "encryptionkey@0"
I20260806 02:45:51.935374 7191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "65f62e6fe4e244f0938a30a6ba603591"
format_stamp: "Formatted at 2026-08-06 02:45:51 on dist-test-slave-8tdl"
server_key: "4e18da3ba30446513872c4d069bce1e7"
server_key_iv: "c5742854028fa9a911540af3ba878ff1"
server_key_version: "encryptionkey@0"
I20260806 02:45:51.940941 7191 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 02:45:51.945058 7205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:51.946802 7191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:51.946988 7191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "65f62e6fe4e244f0938a30a6ba603591"
format_stamp: "Formatted at 2026-08-06 02:45:51 on dist-test-slave-8tdl"
server_key: "4e18da3ba30446513872c4d069bce1e7"
server_key_iv: "c5742854028fa9a911540af3ba878ff1"
server_key_version: "encryptionkey@0"
I20260806 02:45:51.947232 7191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:51.967868 7191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:51.968791 7191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:51.969128 7191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:51.989269 7191 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:39071
I20260806 02:45:51.989302 7256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:39071 every 8 connection(s)
I20260806 02:45:51.990767 7191 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:45:51.994774 7257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:51.995729 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 7191
I20260806 02:45:51.995936 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:45:51.996639 1865 external_mini_cluster.cc:1428] Setting key 6432f011892e6c7b1258eefa4396cbcd
I20260806 02:45:52.011245 7257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65f62e6fe4e244f0938a30a6ba603591: Bootstrap starting.
I20260806 02:45:52.013342 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:51.998390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34754 (local address 127.1.210.126:39071)
0806 02:45:51.999521 (+ 1131us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:51.999533 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:51.999580 (+ 47us) server_negotiation.cc:407] Connection header received
0806 02:45:52.002366 (+ 2786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:52.002397 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:52.003097 (+ 700us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:52.003525 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:52.004216 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:52.005574 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:52.007585 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:52.008659 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:52.008827 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:52.010300 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:52.010310 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:52.010338 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:52.011132 (+ 794us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:52.011232 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:52.011260 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:52.011407 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:52.011729 (+ 322us) server_negotiation.cc:299] Negotiation successful
0806 02:45:52.012284 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":431,"threads_started":1}
I20260806 02:45:52.015019 7257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65f62e6fe4e244f0938a30a6ba603591: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:52.016422 7257 log.cc:824] T 00000000000000000000000000000000 P 65f62e6fe4e244f0938a30a6ba603591: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:52.018036 1865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:45:52.019657 7257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65f62e6fe4e244f0938a30a6ba603591: No bootstrap required, opened a new log
I20260806 02:45:52.028113 7257 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65f62e6fe4e244f0938a30a6ba603591 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f62e6fe4e244f0938a30a6ba603591" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39071 } }
I20260806 02:45:52.028532 7257 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65f62e6fe4e244f0938a30a6ba603591 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:52.028644 7257 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65f62e6fe4e244f0938a30a6ba603591 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65f62e6fe4e244f0938a30a6ba603591, State: Initialized, Role: FOLLOWER
I20260806 02:45:52.029397 7257 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65f62e6fe4e244f0938a30a6ba603591 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f62e6fe4e244f0938a30a6ba603591" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39071 } }
I20260806 02:45:52.029703 7257 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65f62e6fe4e244f0938a30a6ba603591 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:45:52.029855 7257 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65f62e6fe4e244f0938a30a6ba603591 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:45:52.030042 7257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65f62e6fe4e244f0938a30a6ba603591 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:52.032387 7257 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65f62e6fe4e244f0938a30a6ba603591 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f62e6fe4e244f0938a30a6ba603591" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39071 } }
I20260806 02:45:52.032953 7257 leader_election.cc:302] T 00000000000000000000000000000000 P 65f62e6fe4e244f0938a30a6ba603591 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65f62e6fe4e244f0938a30a6ba603591; no voters:
I20260806 02:45:52.033569 7257 leader_election.cc:288] T 00000000000000000000000000000000 P 65f62e6fe4e244f0938a30a6ba603591 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:45:52.033758 7262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65f62e6fe4e244f0938a30a6ba603591 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:52.034543 7262 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65f62e6fe4e244f0938a30a6ba603591 [term 1 LEADER]: Becoming Leader. State: Replica: 65f62e6fe4e244f0938a30a6ba603591, State: Running, Role: LEADER
I20260806 02:45:52.035176 7262 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65f62e6fe4e244f0938a30a6ba603591 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f62e6fe4e244f0938a30a6ba603591" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39071 } }
I20260806 02:45:52.035831 7257 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65f62e6fe4e244f0938a30a6ba603591 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:52.037838 7264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65f62e6fe4e244f0938a30a6ba603591 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65f62e6fe4e244f0938a30a6ba603591. Latest consensus state: current_term: 1 leader_uuid: "65f62e6fe4e244f0938a30a6ba603591" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f62e6fe4e244f0938a30a6ba603591" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39071 } } }
I20260806 02:45:52.038349 7264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65f62e6fe4e244f0938a30a6ba603591 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:52.038730 7263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65f62e6fe4e244f0938a30a6ba603591 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65f62e6fe4e244f0938a30a6ba603591" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f62e6fe4e244f0938a30a6ba603591" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39071 } } }
I20260806 02:45:52.039144 7263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65f62e6fe4e244f0938a30a6ba603591 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:52.040294 7268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:52.045218 7268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:45:52.056644 7268 catalog_manager.cc:1347] Generated new cluster ID: 4953d45af52c489f95a00d88dc32175b
I20260806 02:45:52.056787 7268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:52.081738 7268 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:45:52.083851 7268 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:52.100298 7268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65f62e6fe4e244f0938a30a6ba603591: Generated new TSK 0
I20260806 02:45:52.101547 7268 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 02:45:52.147620 7289 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260806 02:45:52.147778 7289 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 02:45:52.149142 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:52.127332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34768 (local address 127.1.210.126:39071)
0806 02:45:52.127555 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:52.127563 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:52.128229 (+ 666us) server_negotiation.cc:407] Connection header received
0806 02:45:52.128459 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:52.128466 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:52.128563 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:52.128772 (+ 209us) server_negotiation.cc:226] Negotiated authn=JWT
0806 02:45:52.129257 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:52.130458 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:52.132138 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:52.132818 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:52.132952 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:52.133613 (+ 661us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 02:45:52.148356 (+ 14743us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37461/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 02:45:52.148810 (+ 454us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34768: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37461/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":76}
W20260806 02:45:52.149292 7260 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34768: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37461/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260806 02:45:52.149760 7288 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:52.127316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:39071 (local address 127.0.0.1:34768)
0806 02:45:52.128107 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:52.128126 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:52.128268 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:52.128819 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:52.128823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:52.128834 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0806 02:45:52.129099 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:52.129107 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:52.130643 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:52.130648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:52.131896 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:52.131913 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:52.133034 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:52.133039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:52.133205 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:52.133247 (+ 42us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 02:45:52.148685 (+ 15438us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37461/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 02:45:52.149026 (+ 341us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.210.126:39071: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37461/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":442,"threads_started":1}
I20260806 02:45:52.151151 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 7191
I20260806 02:45:52.152585 7252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:45:52.250517 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 7191
2026-08-06T02:45:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (810 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260806 02:45:52.305917 1865 mini_oidc.cc:152] Starting JWKS server
I20260806 02:45:52.308223 1865 webserver.cc:460] Webserver started at https://127.0.0.1:40827/ using document root <none> and password file <none>
I20260806 02:45:52.308533 1865 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 02:45:52.309289 1865 webserver.cc:460] Webserver started at http://127.0.0.1:41631/ using document root <none> and password file <none>
2026-08-06T02:45:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:52Z Disabled control of system clock
I20260806 02:45:52.343659 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:38225
--webserver_interface=127.1.210.126
--webserver_port=0
--builtin_ntp_servers=127.1.210.84:42185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:38225
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40827/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 02:45:52.687830 7296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:52.688225 7296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:52.688388 7296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:52.695961 7296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:52.696134 7296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:52.696224 7296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:52.696327 7296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:52.696409 7296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 02:45:52.696501 7296 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 02:45:52.696573 7296 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40827/jwks.json
I20260806 02:45:52.706415 7296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:42185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:38225
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:38225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40827/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:52.707890 7296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:52.709872 7296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:52.722450 7301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:52.722374 7304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:52.722414 7302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:52.723438 7296 server_base.cc:1034] running on GCE node
I20260806 02:45:52.724288 7296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:52.726150 7296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:52.727440 7296 hybrid_clock.cc:648] HybridClock initialized: now 1785984352727352 us; error 74 us; skew 500 ppm
I20260806 02:45:52.727903 7296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:45:52.736415 7296 webserver.cc:460] Webserver started at http://127.1.210.126:45611/ using document root <none> and password file <none>
I20260806 02:45:52.737277 7296 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:52.737425 7296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:52.737792 7296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:52.741261 7296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "1b3fbcdc566c405e9d21f4efd8815173"
format_stamp: "Formatted at 2026-08-06 02:45:52 on dist-test-slave-8tdl"
server_key: "102745732b0a71c2241e85db08b148f7"
server_key_iv: "73f9d3a3dcf10e95944b91fba7b5433f"
server_key_version: "encryptionkey@0"
I20260806 02:45:52.742225 7296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "1b3fbcdc566c405e9d21f4efd8815173"
format_stamp: "Formatted at 2026-08-06 02:45:52 on dist-test-slave-8tdl"
server_key: "102745732b0a71c2241e85db08b148f7"
server_key_iv: "73f9d3a3dcf10e95944b91fba7b5433f"
server_key_version: "encryptionkey@0"
I20260806 02:45:52.748276 7296 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260806 02:45:52.752315 7310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:52.754015 7296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:45:52.754243 7296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "1b3fbcdc566c405e9d21f4efd8815173"
format_stamp: "Formatted at 2026-08-06 02:45:52 on dist-test-slave-8tdl"
server_key: "102745732b0a71c2241e85db08b148f7"
server_key_iv: "73f9d3a3dcf10e95944b91fba7b5433f"
server_key_version: "encryptionkey@0"
I20260806 02:45:52.754514 7296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:52.783417 7296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:52.784406 7296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:52.784766 7296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:52.807487 7296 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:38225
I20260806 02:45:52.807583 7361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:38225 every 8 connection(s)
I20260806 02:45:52.809053 7296 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:45:52.813027 7362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:52.818095 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 7296
I20260806 02:45:52.818347 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:45:52.819150 1865 external_mini_cluster.cc:1428] Setting key 3a0d6f5901205be80e34aff1229b62dd
I20260806 02:45:52.825423 7362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b3fbcdc566c405e9d21f4efd8815173: Bootstrap starting.
I20260806 02:45:52.829633 7362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b3fbcdc566c405e9d21f4efd8815173: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:52.830852 7362 log.cc:824] T 00000000000000000000000000000000 P 1b3fbcdc566c405e9d21f4efd8815173: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:52.833927 7362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b3fbcdc566c405e9d21f4efd8815173: No bootstrap required, opened a new log
I20260806 02:45:52.835659 7365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:52.821265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59032 (local address 127.1.210.126:38225)
0806 02:45:52.822490 (+ 1225us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:52.822521 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:52.822575 (+ 54us) server_negotiation.cc:407] Connection header received
0806 02:45:52.825083 (+ 2508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:52.825116 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:52.825791 (+ 675us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:52.826282 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:52.826855 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:52.829004 (+ 2149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:52.830525 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:52.831777 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:52.832007 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:52.833184 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:52.833192 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:52.833207 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:52.833768 (+ 561us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:52.833850 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:52.833872 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:52.834022 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:52.834267 (+ 245us) server_negotiation.cc:299] Negotiation successful
0806 02:45:52.834597 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":377,"threads_started":1}
I20260806 02:45:52.839668 1865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:45:52.842170 7362 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b3fbcdc566c405e9d21f4efd8815173 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b3fbcdc566c405e9d21f4efd8815173" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 38225 } }
I20260806 02:45:52.842589 7362 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b3fbcdc566c405e9d21f4efd8815173 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:52.842725 7362 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b3fbcdc566c405e9d21f4efd8815173 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3fbcdc566c405e9d21f4efd8815173, State: Initialized, Role: FOLLOWER
I20260806 02:45:52.843377 7362 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b3fbcdc566c405e9d21f4efd8815173 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b3fbcdc566c405e9d21f4efd8815173" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 38225 } }
I20260806 02:45:52.843648 7362 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b3fbcdc566c405e9d21f4efd8815173 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:45:52.843789 7362 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b3fbcdc566c405e9d21f4efd8815173 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:45:52.843964 7362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b3fbcdc566c405e9d21f4efd8815173 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:52.846264 7362 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b3fbcdc566c405e9d21f4efd8815173 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b3fbcdc566c405e9d21f4efd8815173" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 38225 } }
I20260806 02:45:52.846938 7362 leader_election.cc:302] T 00000000000000000000000000000000 P 1b3fbcdc566c405e9d21f4efd8815173 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b3fbcdc566c405e9d21f4efd8815173; no voters:
I20260806 02:45:52.847620 7362 leader_election.cc:288] T 00000000000000000000000000000000 P 1b3fbcdc566c405e9d21f4efd8815173 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:45:52.847780 7367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b3fbcdc566c405e9d21f4efd8815173 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:52.848554 7367 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b3fbcdc566c405e9d21f4efd8815173 [term 1 LEADER]: Becoming Leader. State: Replica: 1b3fbcdc566c405e9d21f4efd8815173, State: Running, Role: LEADER
I20260806 02:45:52.849110 7367 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b3fbcdc566c405e9d21f4efd8815173 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b3fbcdc566c405e9d21f4efd8815173" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 38225 } }
I20260806 02:45:52.849905 7362 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b3fbcdc566c405e9d21f4efd8815173 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:52.852035 7368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b3fbcdc566c405e9d21f4efd8815173 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b3fbcdc566c405e9d21f4efd8815173" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b3fbcdc566c405e9d21f4efd8815173" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 38225 } } }
I20260806 02:45:52.852447 7368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b3fbcdc566c405e9d21f4efd8815173 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:52.852905 7369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b3fbcdc566c405e9d21f4efd8815173 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b3fbcdc566c405e9d21f4efd8815173. Latest consensus state: current_term: 1 leader_uuid: "1b3fbcdc566c405e9d21f4efd8815173" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b3fbcdc566c405e9d21f4efd8815173" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 38225 } } }
I20260806 02:45:52.853235 7369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b3fbcdc566c405e9d21f4efd8815173 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:52.855400 7374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:52.860642 7374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:45:52.872267 7374 catalog_manager.cc:1347] Generated new cluster ID: 59904a98cac246c7a439c9ca909ad4e4
I20260806 02:45:52.872511 7374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:52.894469 7374 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:45:52.896489 7374 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:52.905917 7374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b3fbcdc566c405e9d21f4efd8815173: Generated new TSK 0
I20260806 02:45:52.906991 7374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:45:52.936801 7365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:52.928015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59044 (local address 127.1.210.126:38225)
0806 02:45:52.928275 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:52.928283 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:52.928692 (+ 409us) server_negotiation.cc:407] Connection header received
0806 02:45:52.928930 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:52.928934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:52.929014 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:52.929155 (+ 141us) server_negotiation.cc:226] Negotiated authn=JWT
0806 02:45:52.929757 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:52.930505 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:52.932699 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:52.933210 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:52.933317 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:52.933963 (+ 646us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 02:45:52.936073 (+ 2110us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40827/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 02:45:52.936456 (+ 383us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59044: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40827/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":73}
W20260806 02:45:52.936950 7365 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59044: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40827/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260806 02:45:52.937103 7392 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:52.927893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:38225 (local address 127.0.0.1:59044)
0806 02:45:52.928583 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:52.928604 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:52.928760 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:52.929206 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:52.929212 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:52.929225 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0806 02:45:52.929559 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:52.929571 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:52.930654 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:52.930660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:52.932508 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:52.932526 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:52.933426 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:52.933434 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:52.933645 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:52.933696 (+ 51us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 02:45:52.936290 (+ 2594us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40827/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 02:45:52.936636 (+ 346us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.210.126:38225: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40827/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":358,"threads_started":1}
I20260806 02:45:52.938238 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 7296
I20260806 02:45:52.939707 7357 generic_service.cc:196] Checking for leaks (request via RPC)
E20260806 02:45:52.955303 7393 webserver.cc:533] Webserver: sslize failed:
E20260806 02:45:52.955448 7393 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 02:45:53.037514 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 7296
2026-08-06T02:45:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (787 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-06T02:45:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:53Z Disabled control of system clock
I20260806 02:45:53.125291 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:36983
--webserver_interface=127.1.210.126
--webserver_port=0
--builtin_ntp_servers=127.1.210.84:44649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:36983
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984304136196-1865-0/testchainca.cert with env {}
W20260806 02:45:53.470288 7398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:53.470623 7398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:53.470798 7398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:53.479496 7398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:53.479631 7398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:53.479678 7398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:53.479743 7398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:53.479792 7398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 02:45:53.479863 7398 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 02:45:53.479908 7398 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260806 02:45:53.491139 7398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:44649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:36983
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:36983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984304136196-1865-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:53.492511 7398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:53.493819 7398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:53.506578 7404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:53.506744 7403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:53.506580 7406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:53.507253 7398 server_base.cc:1034] running on GCE node
I20260806 02:45:53.508126 7398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:53.509646 7398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:53.510886 7398 hybrid_clock.cc:648] HybridClock initialized: now 1785984353510812 us; error 65 us; skew 500 ppm
I20260806 02:45:53.511348 7398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:45:53.513803 7398 webserver.cc:460] Webserver started at http://127.1.210.126:36451/ using document root <none> and password file <none>
I20260806 02:45:53.514547 7398 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:53.514714 7398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:53.515069 7398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:53.518195 7398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "c0c51acaab4d4f0f8b8b6498eb060c2a"
format_stamp: "Formatted at 2026-08-06 02:45:53 on dist-test-slave-8tdl"
server_key: "74553a05934e8c6b4d7dda961449a1de"
server_key_iv: "716f69171e6ba62fe517229bf4b68bb2"
server_key_version: "encryptionkey@0"
I20260806 02:45:53.519135 7398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "c0c51acaab4d4f0f8b8b6498eb060c2a"
format_stamp: "Formatted at 2026-08-06 02:45:53 on dist-test-slave-8tdl"
server_key: "74553a05934e8c6b4d7dda961449a1de"
server_key_iv: "716f69171e6ba62fe517229bf4b68bb2"
server_key_version: "encryptionkey@0"
I20260806 02:45:53.524660 7398 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260806 02:45:53.528666 7412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:53.530332 7398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:45:53.530545 7398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "c0c51acaab4d4f0f8b8b6498eb060c2a"
format_stamp: "Formatted at 2026-08-06 02:45:53 on dist-test-slave-8tdl"
server_key: "74553a05934e8c6b4d7dda961449a1de"
server_key_iv: "716f69171e6ba62fe517229bf4b68bb2"
server_key_version: "encryptionkey@0"
I20260806 02:45:53.530860 7398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:53.549794 7398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:53.550832 7398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:53.551152 7398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:53.571499 7398 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:36983
I20260806 02:45:53.571522 7463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:36983 every 8 connection(s)
I20260806 02:45:53.573005 7398 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:45:53.577200 7464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:53.579125 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 7398
I20260806 02:45:53.579341 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:45:53.579999 1865 external_mini_cluster.cc:1428] Setting key 5e7f102fb964a6416757f0bc3e638bf4
I20260806 02:45:53.590019 7464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0c51acaab4d4f0f8b8b6498eb060c2a: Bootstrap starting.
I20260806 02:45:53.593613 7464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0c51acaab4d4f0f8b8b6498eb060c2a: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:53.594808 7464 log.cc:824] T 00000000000000000000000000000000 P c0c51acaab4d4f0f8b8b6498eb060c2a: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:53.595062 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:53.582119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51012 (local address 127.1.210.126:36983)
0806 02:45:53.583281 (+ 1162us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:53.583293 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:53.583339 (+ 46us) server_negotiation.cc:407] Connection header received
0806 02:45:53.585113 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:53.585141 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:53.585653 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:53.586100 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:53.586799 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:53.588465 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:53.589840 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:53.590620 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:53.590878 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:53.592318 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:53.592327 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:53.592343 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:53.592960 (+ 617us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:53.593060 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:53.593084 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:53.593218 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:53.593470 (+ 252us) server_negotiation.cc:299] Negotiation successful
0806 02:45:53.593896 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":349,"threads_started":1}
I20260806 02:45:53.597883 7464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0c51acaab4d4f0f8b8b6498eb060c2a: No bootstrap required, opened a new log
I20260806 02:45:53.599084 1865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:45:53.606096 7464 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0c51acaab4d4f0f8b8b6498eb060c2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0c51acaab4d4f0f8b8b6498eb060c2a" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 36983 } }
I20260806 02:45:53.606535 7464 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0c51acaab4d4f0f8b8b6498eb060c2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:53.606654 7464 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0c51acaab4d4f0f8b8b6498eb060c2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0c51acaab4d4f0f8b8b6498eb060c2a, State: Initialized, Role: FOLLOWER
I20260806 02:45:53.607317 7464 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0c51acaab4d4f0f8b8b6498eb060c2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0c51acaab4d4f0f8b8b6498eb060c2a" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 36983 } }
I20260806 02:45:53.607549 7464 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0c51acaab4d4f0f8b8b6498eb060c2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:45:53.607700 7464 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0c51acaab4d4f0f8b8b6498eb060c2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:45:53.607867 7464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0c51acaab4d4f0f8b8b6498eb060c2a [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:53.610391 7464 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0c51acaab4d4f0f8b8b6498eb060c2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0c51acaab4d4f0f8b8b6498eb060c2a" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 36983 } }
I20260806 02:45:53.611011 7464 leader_election.cc:302] T 00000000000000000000000000000000 P c0c51acaab4d4f0f8b8b6498eb060c2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0c51acaab4d4f0f8b8b6498eb060c2a; no voters:
I20260806 02:45:53.611757 7464 leader_election.cc:288] T 00000000000000000000000000000000 P c0c51acaab4d4f0f8b8b6498eb060c2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:45:53.611878 7469 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0c51acaab4d4f0f8b8b6498eb060c2a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:53.612654 7469 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0c51acaab4d4f0f8b8b6498eb060c2a [term 1 LEADER]: Becoming Leader. State: Replica: c0c51acaab4d4f0f8b8b6498eb060c2a, State: Running, Role: LEADER
I20260806 02:45:53.613286 7469 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0c51acaab4d4f0f8b8b6498eb060c2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0c51acaab4d4f0f8b8b6498eb060c2a" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 36983 } }
I20260806 02:45:53.614367 7464 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0c51acaab4d4f0f8b8b6498eb060c2a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:53.616742 7471 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0c51acaab4d4f0f8b8b6498eb060c2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0c51acaab4d4f0f8b8b6498eb060c2a. Latest consensus state: current_term: 1 leader_uuid: "c0c51acaab4d4f0f8b8b6498eb060c2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0c51acaab4d4f0f8b8b6498eb060c2a" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 36983 } } }
I20260806 02:45:53.617198 7471 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0c51acaab4d4f0f8b8b6498eb060c2a [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:53.617782 7470 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0c51acaab4d4f0f8b8b6498eb060c2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0c51acaab4d4f0f8b8b6498eb060c2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0c51acaab4d4f0f8b8b6498eb060c2a" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 36983 } } }
I20260806 02:45:53.618129 7470 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0c51acaab4d4f0f8b8b6498eb060c2a [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:53.619068 7475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:53.625171 7475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:45:53.635017 7475 catalog_manager.cc:1347] Generated new cluster ID: 02374570a03d462a86c0637116a96e72
I20260806 02:45:53.635149 7475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:53.647341 7475 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:45:53.648801 7475 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:53.655485 7475 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0c51acaab4d4f0f8b8b6498eb060c2a: Generated new TSK 0
I20260806 02:45:53.656461 7475 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:45:54.165493 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:53.668351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51020 (local address 127.1.210.126:36983)
0806 02:45:53.668523 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:53.668551 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:53.669059 (+ 508us) server_negotiation.cc:407] Connection header received
0806 02:45:53.669338 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:53.669342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:53.669421 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:53.669597 (+ 176us) server_negotiation.cc:226] Negotiated authn=JWT
0806 02:45:53.670160 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:53.671111 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:53.673334 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:53.673893 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:53.674038 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:53.674741 (+ 703us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 02:45:54.164758 (+490017us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 02:45:54.165240 (+ 482us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51020: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":60}
W20260806 02:45:54.165634 7467 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51020: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260806 02:45:54.165855 7493 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:53.668366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:36983 (local address 127.0.0.1:51020)
0806 02:45:53.668938 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:53.668955 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:53.669139 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:53.669610 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:53.669614 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:53.669626 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0806 02:45:53.669934 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:53.669942 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:53.671280 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:53.671287 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:53.673142 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:53.673161 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:53.674106 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:53.674113 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:53.674311 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:53.674371 (+ 60us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 02:45:54.165174 (+490803us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 02:45:54.165426 (+ 252us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.210.126:36983: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":315,"threads_started":1}
I20260806 02:45:54.167234 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 7398
I20260806 02:45:54.168561 7459 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:45:54.274255 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 7398
2026-08-06T02:45:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1219 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (429 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (447 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:45:55 dist-test-slave-8tdl krb5kdc[7508](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:45:55 dist-test-slave-8tdl krb5kdc[7508](info): set up 2 sockets
Aug 06 02:45:55 dist-test-slave-8tdl krb5kdc[7508](info): commencing operation
krb5kdc: starting...
W20260806 02:45:56.303037 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:45:56 dist-test-slave-8tdl krb5kdc[7508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:45:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
I20260806 02:45:56.555374 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:46471
--webserver_interface=127.1.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:38163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:46471
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:56.912675 7522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:56.913162 7522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:56.913354 7522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:56.924971 7522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:56.925091 7522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:56.925138 7522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:56.925212 7522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:56.925257 7522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:45:56.939152 7522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:38163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:46471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:46471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:56.940498 7522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:56.941807 7522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:56.954079 7527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:56.954113 7530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:56.954137 7528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:56.955063 7522 server_base.cc:1034] running on GCE node
I20260806 02:45:56.955915 7522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:56.957445 7522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:56.958737 7522 hybrid_clock.cc:648] HybridClock initialized: now 1785984356958632 us; error 58 us; skew 500 ppm
I20260806 02:45:56.959128 7522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:45:56 dist-test-slave-8tdl krb5kdc[7508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984356, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:56.962646 7522 init.cc:377] Logged in from keytab as kudu/127.1.210.126@KRBTEST.COM (short username kudu)
I20260806 02:45:56.964586 7522 webserver.cc:460] Webserver started at http://127.1.210.126:33869/ using document root <none> and password file <none>
I20260806 02:45:56.965358 7522 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:56.965514 7522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:56.965850 7522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:56.969208 7522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "3ae30618d2c8438b947d32deb1d4e964"
format_stamp: "Formatted at 2026-08-06 02:45:56 on dist-test-slave-8tdl"
server_key: "dd9c65f5e32c7cab286d65f89c0318c0"
server_key_iv: "e8d437a2a79cbc88ddadb2471e07b0e4"
server_key_version: "encryptionkey@0"
I20260806 02:45:56.970042 7522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "3ae30618d2c8438b947d32deb1d4e964"
format_stamp: "Formatted at 2026-08-06 02:45:56 on dist-test-slave-8tdl"
server_key: "dd9c65f5e32c7cab286d65f89c0318c0"
server_key_iv: "e8d437a2a79cbc88ddadb2471e07b0e4"
server_key_version: "encryptionkey@0"
I20260806 02:45:56.976034 7522 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260806 02:45:56.980150 7537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:56.981783 7522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260806 02:45:56.982036 7522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "3ae30618d2c8438b947d32deb1d4e964"
format_stamp: "Formatted at 2026-08-06 02:45:56 on dist-test-slave-8tdl"
server_key: "dd9c65f5e32c7cab286d65f89c0318c0"
server_key_iv: "e8d437a2a79cbc88ddadb2471e07b0e4"
server_key_version: "encryptionkey@0"
I20260806 02:45:56.982299 7522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:57.004994 7522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:57.010352 7522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:57.010743 7522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:57.031472 7522 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:46471
I20260806 02:45:57.031504 7588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:46471 every 8 connection(s)
I20260806 02:45:57.033025 7522 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:45:57.037204 7589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:57.039849 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 7522
I20260806 02:45:57.040099 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:45:57.041102 1865 external_mini_cluster.cc:1428] Setting key f7b64fdfc906568102474fd2b62932ea
I20260806 02:45:57.051585 7589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ae30618d2c8438b947d32deb1d4e964: Bootstrap starting.
I20260806 02:45:57.055593 7589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ae30618d2c8438b947d32deb1d4e964: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:57.056747 7589 log.cc:824] T 00000000000000000000000000000000 P 3ae30618d2c8438b947d32deb1d4e964: Log is configured to *not* fsync() on all Append() calls
Aug 06 02:45:57 dist-test-slave-8tdl krb5kdc[7508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:57.060392 7589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ae30618d2c8438b947d32deb1d4e964: No bootstrap required, opened a new log
I20260806 02:45:57.068578 7592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:57.043606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48958 (local address 127.1.210.126:46471)
0806 02:45:57.044455 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:57.044470 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:57.044525 (+ 55us) server_negotiation.cc:407] Connection header received
0806 02:45:57.046883 (+ 2358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:57.046922 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:57.047571 (+ 649us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:57.048118 (+ 547us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:57.049594 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.051817 (+ 2223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.053235 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.054368 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.054582 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:57.059578 (+ 4996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:57.059588 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:57.059605 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:57.059645 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:57.063382 (+ 3737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:57.064458 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:57.064466 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:57.064502 (+ 36us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:57.064716 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:57.065206 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:57.065210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:57.065212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:57.065934 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:57.066168 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:57.066843 (+ 675us) server_negotiation.cc:299] Negotiation successful
0806 02:45:57.067385 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":364,"threads_started":1}
I20260806 02:45:57.070560 7589 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ae30618d2c8438b947d32deb1d4e964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ae30618d2c8438b947d32deb1d4e964" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 46471 } }
I20260806 02:45:57.071077 7589 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ae30618d2c8438b947d32deb1d4e964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:57.071210 7589 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ae30618d2c8438b947d32deb1d4e964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ae30618d2c8438b947d32deb1d4e964, State: Initialized, Role: FOLLOWER
I20260806 02:45:57.071933 7589 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ae30618d2c8438b947d32deb1d4e964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ae30618d2c8438b947d32deb1d4e964" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 46471 } }
I20260806 02:45:57.072268 7589 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ae30618d2c8438b947d32deb1d4e964 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:45:57.072427 7589 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ae30618d2c8438b947d32deb1d4e964 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:45:57.072613 7589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ae30618d2c8438b947d32deb1d4e964 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:57.075003 7589 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ae30618d2c8438b947d32deb1d4e964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ae30618d2c8438b947d32deb1d4e964" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 46471 } }
I20260806 02:45:57.075628 7589 leader_election.cc:302] T 00000000000000000000000000000000 P 3ae30618d2c8438b947d32deb1d4e964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ae30618d2c8438b947d32deb1d4e964; no voters:
I20260806 02:45:57.076350 7589 leader_election.cc:288] T 00000000000000000000000000000000 P 3ae30618d2c8438b947d32deb1d4e964 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:45:57.076459 7594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ae30618d2c8438b947d32deb1d4e964 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:57.077241 7594 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ae30618d2c8438b947d32deb1d4e964 [term 1 LEADER]: Becoming Leader. State: Replica: 3ae30618d2c8438b947d32deb1d4e964, State: Running, Role: LEADER
I20260806 02:45:57.077903 7594 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ae30618d2c8438b947d32deb1d4e964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ae30618d2c8438b947d32deb1d4e964" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 46471 } }
I20260806 02:45:57.078933 7589 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ae30618d2c8438b947d32deb1d4e964 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:57.081272 7595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ae30618d2c8438b947d32deb1d4e964 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ae30618d2c8438b947d32deb1d4e964" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ae30618d2c8438b947d32deb1d4e964" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 46471 } } }
I20260806 02:45:57.081744 7595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ae30618d2c8438b947d32deb1d4e964 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:57.083214 7596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ae30618d2c8438b947d32deb1d4e964 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ae30618d2c8438b947d32deb1d4e964. Latest consensus state: current_term: 1 leader_uuid: "3ae30618d2c8438b947d32deb1d4e964" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ae30618d2c8438b947d32deb1d4e964" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 46471 } } }
I20260806 02:45:57.083510 7600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:57.083730 7596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ae30618d2c8438b947d32deb1d4e964 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:57.094952 7600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:45:57.108947 7600 catalog_manager.cc:1347] Generated new cluster ID: 093814993bc14dc3a103a270299abe36
I20260806 02:45:57.109097 7600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:57.132795 7600 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:45:57.134936 7600 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:57.151654 7600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ae30618d2c8438b947d32deb1d4e964: Generated new TSK 0
I20260806 02:45:57.152911 7600 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab.
Entry for principal kudu/127.1.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab.
I20260806 02:45:57.220248 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.65:0
--local_ip_for_outbound_sockets=127.1.210.65
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:46471
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:38163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:57.577793 7615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:57.578218 7615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:57.578398 7615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:57.586404 7615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:57.586582 7615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:57.586808 7615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.65
I20260806 02:45:57.597496 7615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:38163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:46471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.65
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:57.599002 7615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:57.600337 7615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:57.614529 7620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:57.614609 7623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:57.614593 7621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:57.615254 7615 server_base.cc:1034] running on GCE node
I20260806 02:45:57.616055 7615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:57.617226 7615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:57.618470 7615 hybrid_clock.cc:648] HybridClock initialized: now 1785984357618420 us; error 43 us; skew 500 ppm
I20260806 02:45:57.618906 7615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:45:57 dist-test-slave-8tdl krb5kdc[7508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984357, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:57.622449 7615 init.cc:377] Logged in from keytab as kudu/127.1.210.65@KRBTEST.COM (short username kudu)
I20260806 02:45:57.624161 7615 webserver.cc:460] Webserver started at http://127.1.210.65:36345/ using document root <none> and password file <none>
I20260806 02:45:57.624915 7615 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:57.625083 7615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:57.625433 7615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:57.628516 7615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-0/data/instance:
uuid: "7cff507dc7b742d8a7cc203b7e9883b4"
format_stamp: "Formatted at 2026-08-06 02:45:57 on dist-test-slave-8tdl"
server_key: "2bde5b58e9cfccc19e42974aa6ba373c"
server_key_iv: "5f04eca8c6319b34ade114e7b366b5c0"
server_key_version: "encryptionkey@0"
I20260806 02:45:57.629349 7615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance:
uuid: "7cff507dc7b742d8a7cc203b7e9883b4"
format_stamp: "Formatted at 2026-08-06 02:45:57 on dist-test-slave-8tdl"
server_key: "2bde5b58e9cfccc19e42974aa6ba373c"
server_key_iv: "5f04eca8c6319b34ade114e7b366b5c0"
server_key_version: "encryptionkey@0"
I20260806 02:45:57.634920 7615 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 02:45:57.638846 7630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:57.640389 7615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:45:57.640595 7615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-0/wal
uuid: "7cff507dc7b742d8a7cc203b7e9883b4"
format_stamp: "Formatted at 2026-08-06 02:45:57 on dist-test-slave-8tdl"
server_key: "2bde5b58e9cfccc19e42974aa6ba373c"
server_key_iv: "5f04eca8c6319b34ade114e7b366b5c0"
server_key_version: "encryptionkey@0"
I20260806 02:45:57.640831 7615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:57.678349 7615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:57.683670 7615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:57.684026 7615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:57.685170 7615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:57.687031 7615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:57.687125 7615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:57.687273 7615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:57.687327 7615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:57.725289 7615 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.65:45531
I20260806 02:45:57.725409 7743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.65:45531 every 8 connection(s)
I20260806 02:45:57.727149 7615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
I20260806 02:45:57.737349 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 7615
I20260806 02:45:57.737638 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance
I20260806 02:45:57.738353 1865 external_mini_cluster.cc:1428] Setting key 01f47172c3e5e6ebb468bd608c901d16
Aug 06 02:45:57 dist-test-slave-8tdl krb5kdc[7508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984357, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:57.756937 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:57.730525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:57955 (local address 127.1.210.126:46471)
0806 02:45:57.731181 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:57.731186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:57.731978 (+ 792us) server_negotiation.cc:407] Connection header received
0806 02:45:57.734059 (+ 2081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:57.734065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:57.734182 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:57.734327 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:57.736212 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.737019 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.739203 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.739677 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.739795 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:57.748716 (+ 8921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:57.748725 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:57.748732 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:57.748788 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:57.752825 (+ 4037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:57.753945 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:57.753955 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:57.753961 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:57.754156 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:57.754657 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:57.754665 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:57.754671 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:57.755130 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:57.755311 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:57.756163 (+ 852us) server_negotiation.cc:299] Negotiation successful
0806 02:45:57.756442 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":351,"threads_started":1}
I20260806 02:45:57.757758 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:57.731006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:46471 (local address 127.1.210.65:57955)
0806 02:45:57.731789 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:57.731839 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:57.733725 (+ 1886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:57.734619 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:57.734629 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:57.735334 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:57.736000 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:57.736019 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.737225 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.737239 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:57.738969 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:57.739008 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.739945 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.739954 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:57.740305 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:57.745655 (+ 5350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:57.745687 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:57.748347 (+ 2660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:57.753095 (+ 4748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:57.753105 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:57.753123 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:57.753741 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:57.754295 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:57.754298 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:57.754301 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:57.754516 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:57.755275 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:57.755282 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:57.755694 (+ 412us) client_negotiation.cc:769] Sending connection context
0806 02:45:57.755971 (+ 277us) client_negotiation.cc:240] Negotiation successful
0806 02:45:57.756462 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":306,"threads_started":1}
I20260806 02:45:57.759784 7744 heartbeater.cc:347] Connected to a master server at 127.1.210.126:46471
I20260806 02:45:57.760156 7744 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:57.761004 7744 heartbeater.cc:511] Master 127.1.210.126:46471 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.210.66@KRBTEST.COM; defaulting to no policy
I20260806 02:45:57.763551 7554 ts_manager.cc:194] Registered new tserver with Master: 7cff507dc7b742d8a7cc203b7e9883b4 (127.1.210.65:45531)
I20260806 02:45:57.765443 7554 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.65@KRBTEST.COM'} at 127.1.210.65:57955
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab.
Entry for principal kudu/127.1.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab.
I20260806 02:45:57.799655 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.66:0
--local_ip_for_outbound_sockets=127.1.210.66
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:46471
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:38163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:58.147740 7750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:58.148044 7750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:58.148140 7750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:58.155866 7750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:58.155992 7750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:58.156111 7750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.66
I20260806 02:45:58.166400 7750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:38163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:46471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.66
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:58.167821 7750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:58.169201 7750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:58.183915 7758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:58.183949 7755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:58.184134 7756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:58.184830 7750 server_base.cc:1034] running on GCE node
I20260806 02:45:58.185539 7750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:58.186594 7750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:58.187889 7750 hybrid_clock.cc:648] HybridClock initialized: now 1785984358187829 us; error 35 us; skew 500 ppm
I20260806 02:45:58.188447 7750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:45:58 dist-test-slave-8tdl krb5kdc[7508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984358, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:58.192178 7750 init.cc:377] Logged in from keytab as kudu/127.1.210.66@KRBTEST.COM (short username kudu)
I20260806 02:45:58.193776 7750 webserver.cc:460] Webserver started at http://127.1.210.66:39721/ using document root <none> and password file <none>
I20260806 02:45:58.194504 7750 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:58.194614 7750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:58.194995 7750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:58.198185 7750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-1/data/instance:
uuid: "812ac5372625434398c5c29782931d9a"
format_stamp: "Formatted at 2026-08-06 02:45:58 on dist-test-slave-8tdl"
server_key: "a0af2bb0ada2d6f9f2c4474cd15f3b66"
server_key_iv: "c4b9565356438562c28dd4e0c4d9ab77"
server_key_version: "encryptionkey@0"
I20260806 02:45:58.199138 7750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance:
uuid: "812ac5372625434398c5c29782931d9a"
format_stamp: "Formatted at 2026-08-06 02:45:58 on dist-test-slave-8tdl"
server_key: "a0af2bb0ada2d6f9f2c4474cd15f3b66"
server_key_iv: "c4b9565356438562c28dd4e0c4d9ab77"
server_key_version: "encryptionkey@0"
I20260806 02:45:58.204840 7750 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 02:45:58.209400 7765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:58.211122 7750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:45:58.211328 7750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-1/wal
uuid: "812ac5372625434398c5c29782931d9a"
format_stamp: "Formatted at 2026-08-06 02:45:58 on dist-test-slave-8tdl"
server_key: "a0af2bb0ada2d6f9f2c4474cd15f3b66"
server_key_iv: "c4b9565356438562c28dd4e0c4d9ab77"
server_key_version: "encryptionkey@0"
I20260806 02:45:58.211558 7750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:58.227864 7750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:58.232831 7750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:58.233163 7750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:58.234268 7750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:58.235824 7750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:58.235914 7750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:58.236054 7750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:58.236123 7750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:58.271207 7750 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.66:46485
I20260806 02:45:58.271227 7878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.66:46485 every 8 connection(s)
I20260806 02:45:58.272590 7750 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
I20260806 02:45:58.277846 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 7750
I20260806 02:45:58.278029 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance
I20260806 02:45:58.278606 1865 external_mini_cluster.cc:1428] Setting key 8a85019a8788fcd3d8ee6d66fb75114c
Aug 06 02:45:58 dist-test-slave-8tdl krb5kdc[7508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984358, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:58.297534 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:58.275358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:43339 (local address 127.1.210.126:46471)
0806 02:45:58.275862 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:58.275867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:58.276749 (+ 882us) server_negotiation.cc:407] Connection header received
0806 02:45:58.279414 (+ 2665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:58.279419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:58.279525 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:58.279650 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:58.281489 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:58.282240 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:58.284964 (+ 2724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:58.285397 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:58.285498 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:58.290320 (+ 4822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:58.290329 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:58.290335 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:58.290383 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:58.293323 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:58.294483 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:58.294488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:58.294491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:58.294658 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:58.295349 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:58.295353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:58.295356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:58.295690 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:58.295841 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:58.296937 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0806 02:45:58.297157 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":287,"threads_started":1}
I20260806 02:45:58.299154 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:58.275751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:46471 (local address 127.1.210.66:43339)
0806 02:45:58.276511 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:58.276561 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:58.279052 (+ 2491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:58.279914 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:58.279923 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:58.280473 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:58.281283 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:58.281312 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:58.282426 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:58.282434 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:58.284791 (+ 2357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:58.284811 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:58.285649 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:58.285659 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:58.286015 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:58.286959 (+ 944us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:58.286996 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:58.290018 (+ 3022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:58.293504 (+ 3486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:58.293517 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:58.293541 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:58.294296 (+ 755us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:58.294902 (+ 606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:58.294910 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:58.294917 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:58.295212 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:58.295850 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:58.295862 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:58.296397 (+ 535us) client_negotiation.cc:769] Sending connection context
0806 02:45:58.296741 (+ 344us) client_negotiation.cc:240] Negotiation successful
0806 02:45:58.297456 (+ 715us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":306,"threads_started":1}
I20260806 02:45:58.300997 7879 heartbeater.cc:347] Connected to a master server at 127.1.210.126:46471
I20260806 02:45:58.301333 7879 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:58.302214 7879 heartbeater.cc:511] Master 127.1.210.126:46471 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.210.67@KRBTEST.COM; defaulting to no policy
I20260806 02:45:58.303977 7554 ts_manager.cc:194] Registered new tserver with Master: 812ac5372625434398c5c29782931d9a (127.1.210.66:46485)
I20260806 02:45:58.304942 7554 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.66@KRBTEST.COM'} at 127.1.210.66:43339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab.
Entry for principal kudu/127.1.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab.
I20260806 02:45:58.336335 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.67:0
--local_ip_for_outbound_sockets=127.1.210.67
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:46471
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:38163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:58.680397 7885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:58.680711 7885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:58.680806 7885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:58.688386 7885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:58.688504 7885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:58.688624 7885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.67
I20260806 02:45:58.698830 7885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:38163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:46471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.67
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:45:58.700165 7885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:58.701454 7885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:58.715837 7890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:58.715996 7891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:58.716202 7893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:58.717293 7885 server_base.cc:1034] running on GCE node
I20260806 02:45:58.718003 7885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:58.719190 7885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:58.720428 7885 hybrid_clock.cc:648] HybridClock initialized: now 1785984358720360 us; error 34 us; skew 500 ppm
I20260806 02:45:58.720847 7885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:45:58 dist-test-slave-8tdl krb5kdc[7508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984358, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:58.724483 7885 init.cc:377] Logged in from keytab as kudu/127.1.210.67@KRBTEST.COM (short username kudu)
I20260806 02:45:58.726146 7885 webserver.cc:460] Webserver started at http://127.1.210.67:37891/ using document root <none> and password file <none>
I20260806 02:45:58.726946 7885 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:58.727085 7885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:58.727416 7885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:58.730748 7885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-2/data/instance:
uuid: "26ddb0d1e1e943898d92e0bb2d425420"
format_stamp: "Formatted at 2026-08-06 02:45:58 on dist-test-slave-8tdl"
server_key: "6a6642b9d9ba78462f2ec03fccb07e75"
server_key_iv: "0c9dacceadaa163cba3eda0049160000"
server_key_version: "encryptionkey@0"
I20260806 02:45:58.731657 7885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance:
uuid: "26ddb0d1e1e943898d92e0bb2d425420"
format_stamp: "Formatted at 2026-08-06 02:45:58 on dist-test-slave-8tdl"
server_key: "6a6642b9d9ba78462f2ec03fccb07e75"
server_key_iv: "0c9dacceadaa163cba3eda0049160000"
server_key_version: "encryptionkey@0"
I20260806 02:45:58.737349 7885 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260806 02:45:58.741370 7900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:58.743021 7885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:58.743216 7885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-2/wal
uuid: "26ddb0d1e1e943898d92e0bb2d425420"
format_stamp: "Formatted at 2026-08-06 02:45:58 on dist-test-slave-8tdl"
server_key: "6a6642b9d9ba78462f2ec03fccb07e75"
server_key_iv: "0c9dacceadaa163cba3eda0049160000"
server_key_version: "encryptionkey@0"
I20260806 02:45:58.743445 7885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:58.765538 7885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:58.768992 7744 heartbeater.cc:503] Master 127.1.210.126:46471 was elected leader, sending a full tablet report...
I20260806 02:45:58.770974 7885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:58.771289 7885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:58.772384 7885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:58.773916 7885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:58.773999 7885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:58.774140 7885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:58.774194 7885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:58.809949 7885 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.67:40687
I20260806 02:45:58.810098 8013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.67:40687 every 8 connection(s)
I20260806 02:45:58.811408 7885 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
I20260806 02:45:58.821028 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 7885
I20260806 02:45:58.821231 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance
I20260806 02:45:58.821907 1865 external_mini_cluster.cc:1428] Setting key 404c6893f390526c0504ea15e69a545f
Aug 06 02:45:58 dist-test-slave-8tdl krb5kdc[7508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984358, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:45:58.836560 8017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:58.816457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:54037 (local address 127.1.210.126:46471)
0806 02:45:58.817033 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:58.817041 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:58.817263 (+ 222us) server_negotiation.cc:407] Connection header received
0806 02:45:58.819473 (+ 2210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:58.819479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:58.819593 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:58.819727 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:58.821946 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:58.822938 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:58.825326 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:58.825731 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:58.825832 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:58.829840 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:58.829845 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:58.829849 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:58.829886 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:58.832739 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:58.833639 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:58.833644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:58.833646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:58.833788 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:58.834312 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:58.834316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:58.834318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:58.834636 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:58.834858 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:58.835756 (+ 898us) server_negotiation.cc:299] Negotiation successful
0806 02:45:58.836094 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":292,"threads_started":1}
I20260806 02:45:58.837628 8016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:58.815900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:46471 (local address 127.1.210.67:54037)
0806 02:45:58.817062 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:58.817118 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:58.819148 (+ 2030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:58.820041 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:58.820050 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:58.820773 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:58.821729 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:58.821759 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:58.823136 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:58.823140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:58.825150 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:58.825164 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:58.825996 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:58.826003 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:58.826236 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:58.827057 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:58.827091 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:58.829590 (+ 2499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:58.832897 (+ 3307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:58.832906 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:58.832924 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:58.833490 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:58.833908 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:58.833911 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:58.833914 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:58.834161 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:58.834842 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:58.834849 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:58.835279 (+ 430us) client_negotiation.cc:769] Sending connection context
0806 02:45:58.835584 (+ 305us) client_negotiation.cc:240] Negotiation successful
0806 02:45:58.836129 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":339,"threads_started":1}
I20260806 02:45:58.839411 8014 heartbeater.cc:347] Connected to a master server at 127.1.210.126:46471
I20260806 02:45:58.839728 8014 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:58.840523 8014 heartbeater.cc:511] Master 127.1.210.126:46471 requested a full tablet report, sending...
I20260806 02:45:58.842298 7554 ts_manager.cc:194] Registered new tserver with Master: 26ddb0d1e1e943898d92e0bb2d425420 (127.1.210.67:40687)
I20260806 02:45:58.843245 7554 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.67@KRBTEST.COM'} at 127.1.210.67:54037
I20260806 02:45:58.852108 1865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/test-admin.keytab.
Aug 06 02:45:58 dist-test-slave-8tdl krb5kdc[7508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:58.885778 1865 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260806 02:45:58.891943 8022 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:58.888853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60072 (local address 127.1.210.65:45531)
0806 02:45:58.889572 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:58.889583 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:58.889674 (+ 91us) server_negotiation.cc:407] Connection header received
0806 02:45:58.889901 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:58.889909 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:58.890195 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:58.890372 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:58.891476 (+ 1104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60072: BlockingRecv error: recv got EOF from 127.0.0.1:60072 (error 108)
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":312,"threads_started":1}
W20260806 02:45:58.892421 1865 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/krb5cc' of type FILE
Aug 06 02:45:58 dist-test-slave-8tdl krb5kdc[7508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:58.894541 1865 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 02:45:58 dist-test-slave-8tdl krb5kdc[7508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:58.896152 1865 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 02:45:58 dist-test-slave-8tdl krb5kdc[7508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.65@KRBTEST.COM
I20260806 02:45:58.914616 8022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:58.897926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60086 (local address 127.1.210.65:45531)
0806 02:45:58.898206 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:58.898213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:58.898555 (+ 342us) server_negotiation.cc:407] Connection header received
0806 02:45:58.898844 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:58.898848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:58.898966 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:58.899064 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:58.900420 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:58.901492 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:58.903036 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:58.903705 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:58.903818 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:58.908254 (+ 4436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:58.908269 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:58.908285 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:58.908334 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:58.911589 (+ 3255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:58.912387 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:58.912396 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:58.912400 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:58.912532 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:58.912971 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:58.912975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:58.912977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:58.913342 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:58.913588 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:58.914012 (+ 424us) server_negotiation.cc:299] Negotiation successful
0806 02:45:58.914308 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260806 02:45:58.921401 8022 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:58.919503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60094 (local address 127.1.210.65:45531)
0806 02:45:58.919721 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:58.919729 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:58.919947 (+ 218us) server_negotiation.cc:407] Connection header received
0806 02:45:58.920015 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:58.920019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:58.920087 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:58.920199 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:58.921099 (+ 900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60094: BlockingRecv error: recv got EOF from 127.0.0.1:60094 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260806 02:45:58.922132 1865 init.cc:244] Error while opening credential cache '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/krb5cc)
Aug 06 02:45:58 dist-test-slave-8tdl krb5kdc[7508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:58.923974 1865 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 02:45:58 dist-test-slave-8tdl krb5kdc[7508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:58.925529 1865 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 02:45:58 dist-test-slave-8tdl krb5kdc[7508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.65@KRBTEST.COM
I20260806 02:45:58.941507 8022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:58.927189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60098 (local address 127.1.210.65:45531)
0806 02:45:58.927428 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:58.927434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:58.927654 (+ 220us) server_negotiation.cc:407] Connection header received
0806 02:45:58.927803 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:58.927807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:58.927869 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:58.928003 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:58.928934 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:58.929724 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:58.931108 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:58.931562 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:58.931671 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:58.935625 (+ 3954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:58.935630 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:58.935634 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:58.935690 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:58.938540 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:58.939364 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:58.939369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:58.939371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:58.939530 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:58.939925 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:58.939938 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:58.939940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:58.940313 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:58.940583 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:58.940884 (+ 301us) server_negotiation.cc:299] Negotiation successful
0806 02:45:58.941211 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260806 02:45:58.946628 8022 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:58.944808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60102 (local address 127.1.210.65:45531)
0806 02:45:58.945002 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:58.945009 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:58.945172 (+ 163us) server_negotiation.cc:407] Connection header received
0806 02:45:58.945301 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:58.945305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:58.945412 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:58.945502 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:58.946365 (+ 863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60102: BlockingRecv error: recv got EOF from 127.0.0.1:60102 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260806 02:45:58.946780 1865 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984304136196-1865-0/krb5kdc/krb5cc' of type FILE
Aug 06 02:45:58 dist-test-slave-8tdl krb5kdc[7508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:58.948421 1865 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 02:45:58 dist-test-slave-8tdl krb5kdc[7508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:58.949868 1865 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 02:45:58 dist-test-slave-8tdl krb5kdc[7508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.65@KRBTEST.COM
I20260806 02:45:58.965281 8022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:58.951448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60108 (local address 127.1.210.65:45531)
0806 02:45:58.951684 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:58.951690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:58.951817 (+ 127us) server_negotiation.cc:407] Connection header received
0806 02:45:58.951960 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:58.951964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:58.952022 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:58.952101 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:58.953016 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:58.953760 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:58.955049 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:58.955492 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:58.955593 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:58.959546 (+ 3953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:58.959552 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:58.959556 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:58.959592 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:58.962466 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:58.963277 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:58.963281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:58.963283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:58.963428 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:58.963794 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:58.963798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:58.963801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:58.964180 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:58.964443 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:58.964674 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 02:45:58.964986 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 02:45:58.967109 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 7615
I20260806 02:45:58.981086 8022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:58.967796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60124 (local address 127.1.210.65:45531)
0806 02:45:58.967985 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:58.967989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:58.968348 (+ 359us) server_negotiation.cc:407] Connection header received
0806 02:45:58.968466 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:58.968470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:58.968528 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:58.968612 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:58.969662 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:58.970461 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:58.971723 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:58.972173 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:58.972289 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:58.974341 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:58.974346 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:58.974350 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:58.974380 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:58.977312 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:58.978250 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:58.978254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:58.978256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:58.978453 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:58.979180 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:58.979189 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:58.979193 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:58.979735 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:58.980122 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:58.980467 (+ 345us) server_negotiation.cc:299] Negotiation successful
0806 02:45:58.980687 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 02:45:58.981849 7739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:45:59.075475 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 7615
I20260806 02:45:59.093382 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 7750
Aug 06 02:45:59 dist-test-slave-8tdl krb5kdc[7508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.66@KRBTEST.COM
I20260806 02:45:59.112479 8035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:59.094426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53828 (local address 127.1.210.66:46485)
0806 02:45:59.095170 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:59.095179 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:59.095207 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:45:59.095269 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:59.095276 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:59.095499 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:59.095655 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:59.096714 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:59.097877 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:59.099602 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:59.100252 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:59.100376 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:59.105061 (+ 4685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:59.105079 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:59.105095 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:59.105140 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:59.108512 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:59.109427 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:59.109436 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:59.109438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:59.109630 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:59.110197 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:59.110201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:59.110204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:59.111092 (+ 888us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:59.111360 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:59.111777 (+ 417us) server_negotiation.cc:299] Negotiation successful
0806 02:45:59.112154 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":362,"threads_started":1}
I20260806 02:45:59.113823 7874 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:45:59.199116 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 7750
I20260806 02:45:59.215798 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 7885
Aug 06 02:45:59 dist-test-slave-8tdl krb5kdc[7508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.67@KRBTEST.COM
I20260806 02:45:59.234112 8037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:59.216849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33460 (local address 127.1.210.67:40687)
0806 02:45:59.217559 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:59.217566 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:59.217598 (+ 32us) server_negotiation.cc:407] Connection header received
0806 02:45:59.217658 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:59.217665 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:59.217883 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:59.218035 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:59.219279 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:59.220385 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:59.221782 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:59.222432 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:59.222545 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:59.226960 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:59.226990 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:59.227012 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:59.227077 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:59.230834 (+ 3757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:59.231820 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:59.231829 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:59.231832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:59.232056 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:59.232533 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:59.232538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:59.232540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:59.232991 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:59.233273 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:59.233571 (+ 298us) server_negotiation.cc:299] Negotiation successful
0806 02:45:59.233797 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":395,"threads_started":1}
I20260806 02:45:59.235683 8009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:45:59.321396 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 7885
I20260806 02:45:59.339293 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 7522
I20260806 02:45:59.340587 7584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:45:59.445897 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 7522
2026-08-06T02:45:59Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4299 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:45:59 dist-test-slave-8tdl krb5kdc[8042](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:45:59 dist-test-slave-8tdl krb5kdc[8042](info): set up 2 sockets
Aug 06 02:45:59 dist-test-slave-8tdl krb5kdc[8042](info): commencing operation
krb5kdc: starting...
W20260806 02:46:00.575107 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:46:00 dist-test-slave-8tdl krb5kdc[8042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:46:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:46:00Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.126.keytab.
Entry for principal oryx/127.1.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.126.keytab.
I20260806 02:46:00.823419 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:41363
--webserver_interface=127.1.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.126.keytab
--principal=oryx/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:42477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:41363
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:46:01.170275 8055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:01.170693 8055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:01.170883 8055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:01.178344 8055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:46:01.178510 8055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:01.178629 8055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:01.178766 8055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:46:01.178853 8055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:46:01.189692 8055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:42477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:41363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.126.keytab
--principal=oryx/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:41363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:46:01.191251 8055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:01.192713 8055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:01.205256 8060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:01.205353 8063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:01.205319 8061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:01.208518 8055 server_base.cc:1034] running on GCE node
I20260806 02:46:01.209424 8055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:01.211078 8055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:01.212357 8055 hybrid_clock.cc:648] HybridClock initialized: now 1785984361212309 us; error 55 us; skew 500 ppm
I20260806 02:46:01.212779 8055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:46:01 dist-test-slave-8tdl krb5kdc[8042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984361, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:01.216497 8055 init.cc:377] Logged in from keytab as oryx/127.1.210.126@KRBTEST.COM (short username oryx)
I20260806 02:46:01.218243 8055 webserver.cc:460] Webserver started at http://127.1.210.126:37147/ using document root <none> and password file <none>
I20260806 02:46:01.219193 8055 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:01.219386 8055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:01.219750 8055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:01.223268 8055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "869bf3f6f6ad40e5ba34f73d3004a761"
format_stamp: "Formatted at 2026-08-06 02:46:01 on dist-test-slave-8tdl"
server_key: "5d7f8271d87ffd62efd68227acfc8030"
server_key_iv: "7e08db97906d55f22f896a663238cae6"
server_key_version: "encryptionkey@0"
I20260806 02:46:01.224185 8055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "869bf3f6f6ad40e5ba34f73d3004a761"
format_stamp: "Formatted at 2026-08-06 02:46:01 on dist-test-slave-8tdl"
server_key: "5d7f8271d87ffd62efd68227acfc8030"
server_key_iv: "7e08db97906d55f22f896a663238cae6"
server_key_version: "encryptionkey@0"
I20260806 02:46:01.230235 8055 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260806 02:46:01.234269 8070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:01.236112 8055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:46:01.236320 8055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "869bf3f6f6ad40e5ba34f73d3004a761"
format_stamp: "Formatted at 2026-08-06 02:46:01 on dist-test-slave-8tdl"
server_key: "5d7f8271d87ffd62efd68227acfc8030"
server_key_iv: "7e08db97906d55f22f896a663238cae6"
server_key_version: "encryptionkey@0"
I20260806 02:46:01.236536 8055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:01.258056 8055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:01.262845 8055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:01.263162 8055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:01.282492 8055 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:41363
I20260806 02:46:01.282527 8123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:41363 every 8 connection(s)
I20260806 02:46:01.284009 8055 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:46:01.287772 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 8055
I20260806 02:46:01.288017 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:46:01.288540 8124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:01.288867 1865 external_mini_cluster.cc:1428] Setting key 7755a85bf255d748c5fca80d86d6aa1a
I20260806 02:46:01.304040 8124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761: Bootstrap starting.
Aug 06 02:46:01 dist-test-slave-8tdl krb5kdc[8042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM
I20260806 02:46:01.308337 8124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:01.309774 8124 log.cc:824] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761: Log is configured to *not* fsync() on all Append() calls
I20260806 02:46:01.313475 8124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761: No bootstrap required, opened a new log
I20260806 02:46:01.314783 8127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:01.291051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56372 (local address 127.1.210.126:41363)
0806 02:46:01.291935 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:01.291947 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:01.291990 (+ 43us) server_negotiation.cc:407] Connection header received
0806 02:46:01.293566 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:01.293588 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:01.293999 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:01.294335 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:01.296274 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:01.297669 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:01.299494 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:01.300288 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:01.300442 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:01.305472 (+ 5030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:01.305483 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:01.305506 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:01.305574 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:01.309749 (+ 4175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:01.310807 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:01.310814 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:01.310821 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:01.310994 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:01.311442 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:01.311447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:01.311449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:01.312174 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:01.312500 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:01.312853 (+ 353us) server_negotiation.cc:299] Negotiation successful
0806 02:46:01.313315 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":397,"threads_started":1}
I20260806 02:46:01.322788 8124 raft_consensus.cc:348] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } }
I20260806 02:46:01.323217 8124 raft_consensus.cc:374] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:01.323328 8124 raft_consensus.cc:729] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869bf3f6f6ad40e5ba34f73d3004a761, State: Initialized, Role: FOLLOWER
I20260806 02:46:01.323937 8124 consensus_queue.cc:260] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } }
I20260806 02:46:01.324193 8124 raft_consensus.cc:388] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:46:01.324334 8124 raft_consensus.cc:482] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:46:01.324510 8124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:01.326856 8124 raft_consensus.cc:504] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } }
I20260806 02:46:01.327386 8124 leader_election.cc:302] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 869bf3f6f6ad40e5ba34f73d3004a761; no voters:
I20260806 02:46:01.328044 8124 leader_election.cc:288] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:46:01.328274 8129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:46:01.329491 8129 raft_consensus.cc:686] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [term 1 LEADER]: Becoming Leader. State: Replica: 869bf3f6f6ad40e5ba34f73d3004a761, State: Running, Role: LEADER
I20260806 02:46:01.330077 8129 consensus_queue.cc:237] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } }
I20260806 02:46:01.330331 8124 sys_catalog.cc:564] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:46:01.333307 8130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } }
I20260806 02:46:01.333724 8130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [sys.catalog]: This master's current role is: LEADER
I20260806 02:46:01.333379 8131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 869bf3f6f6ad40e5ba34f73d3004a761. Latest consensus state: current_term: 1 leader_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } }
I20260806 02:46:01.333848 8131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [sys.catalog]: This master's current role is: LEADER
I20260806 02:46:01.344033 8137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:46:01.349649 8137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:46:01.359225 8137 catalog_manager.cc:1347] Generated new cluster ID: fa59af352fb5431abc8784c5d36b6618
I20260806 02:46:01.359359 8137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:46:01.371894 8137 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:46:01.373834 8137 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:46:01.384917 8137 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761: Generated new TSK 0
I20260806 02:46:01.386216 8137 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.65.keytab.
Entry for principal oryx/127.1.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.65.keytab.
I20260806 02:46:01.458920 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.65:0
--local_ip_for_outbound_sockets=127.1.210.65
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:41363
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.65.keytab
--principal=oryx/127.1.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:42477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:46:01.810456 8150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:01.810910 8150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:01.811092 8150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:01.818871 8150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:01.819023 8150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:01.819207 8150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.65
I20260806 02:46:01.830886 8150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:42477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.65.keytab
--principal=oryx/127.1.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:41363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.65
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:46:01.832373 8150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:01.833801 8150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:01.847873 8155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:01.847918 8156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:01.847831 8158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:01.849236 8150 server_base.cc:1034] running on GCE node
I20260806 02:46:01.849993 8150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:01.851297 8150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:01.852571 8150 hybrid_clock.cc:648] HybridClock initialized: now 1785984361852522 us; error 61 us; skew 500 ppm
I20260806 02:46:01.852983 8150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:46:01 dist-test-slave-8tdl krb5kdc[8042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984361, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:01.856570 8150 init.cc:377] Logged in from keytab as oryx/127.1.210.65@KRBTEST.COM (short username oryx)
I20260806 02:46:01.858116 8150 webserver.cc:460] Webserver started at http://127.1.210.65:33851/ using document root <none> and password file <none>
I20260806 02:46:01.858871 8150 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:01.858986 8150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:01.859258 8150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:01.862391 8150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-0/data/instance:
uuid: "cea96d710f5c430db2f412a731bd1bee"
format_stamp: "Formatted at 2026-08-06 02:46:01 on dist-test-slave-8tdl"
server_key: "7bb7675311034e5f40114255e889cd17"
server_key_iv: "f4a86902c1ef64f69b2da72bacdee46a"
server_key_version: "encryptionkey@0"
I20260806 02:46:01.863221 8150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance:
uuid: "cea96d710f5c430db2f412a731bd1bee"
format_stamp: "Formatted at 2026-08-06 02:46:01 on dist-test-slave-8tdl"
server_key: "7bb7675311034e5f40114255e889cd17"
server_key_iv: "f4a86902c1ef64f69b2da72bacdee46a"
server_key_version: "encryptionkey@0"
I20260806 02:46:01.868672 8150 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 02:46:01.872437 8165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:01.874140 8150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:46:01.874284 8150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-0/wal
uuid: "cea96d710f5c430db2f412a731bd1bee"
format_stamp: "Formatted at 2026-08-06 02:46:01 on dist-test-slave-8tdl"
server_key: "7bb7675311034e5f40114255e889cd17"
server_key_iv: "f4a86902c1ef64f69b2da72bacdee46a"
server_key_version: "encryptionkey@0"
I20260806 02:46:01.874459 8150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:01.905211 8150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:01.909961 8150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:01.910219 8150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:01.911305 8150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:46:01.912711 8150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:01.912793 8150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:01.912880 8150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:01.912930 8150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:01.953939 8150 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.65:42461
I20260806 02:46:01.953982 8278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.65:42461 every 8 connection(s)
I20260806 02:46:01.955384 8150 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
I20260806 02:46:01.955920 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 8150
I20260806 02:46:01.956184 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance
I20260806 02:46:01.957074 1865 external_mini_cluster.cc:1428] Setting key 519d4d793b2964756a3b687fc2a3e73d
Aug 06 02:46:01 dist-test-slave-8tdl krb5kdc[8042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984361, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM
I20260806 02:46:01.978210 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:01.958358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:42951 (local address 127.1.210.126:41363)
0806 02:46:01.958896 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:01.958903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:01.959935 (+ 1032us) server_negotiation.cc:407] Connection header received
0806 02:46:01.961795 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:01.961801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:01.961906 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:01.962031 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:01.963724 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:01.964452 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:01.966045 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:01.966462 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:01.966600 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:01.970537 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:01.970547 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:01.970556 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:01.970605 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:01.974249 (+ 3644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:01.975161 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:01.975170 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:01.975176 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:01.975348 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:01.975809 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:01.975818 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:01.975824 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:01.976237 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:01.976397 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:01.977390 (+ 993us) server_negotiation.cc:299] Negotiation successful
0806 02:46:01.977676 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":289,"threads_started":1}
I20260806 02:46:01.979516 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:01.958902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.1.210.65:42951)
0806 02:46:01.959777 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:01.959814 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:01.961513 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:01.962290 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:01.962300 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:01.962921 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:01.963539 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:01.963558 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:01.964624 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:01.964628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:01.965888 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:01.965900 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:01.966737 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:01.966745 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:01.966978 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:01.967736 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:01.967760 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:01.970274 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:01.974421 (+ 4147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:01.974429 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:01.974443 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:01.975015 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:01.975468 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:01.975471 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:01.975474 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:01.975672 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:01.976410 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:01.976420 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:01.976840 (+ 420us) client_negotiation.cc:769] Sending connection context
0806 02:46:01.977216 (+ 376us) client_negotiation.cc:240] Negotiation successful
0806 02:46:01.977753 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":356,"threads_started":1}
WARNING: no policy specified for oryx/127.1.210.66@KRBTEST.COM; defaulting to no policy
I20260806 02:46:01.981489 8279 heartbeater.cc:347] Connected to a master server at 127.1.210.126:41363
I20260806 02:46:01.981803 8279 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:01.982581 8279 heartbeater.cc:511] Master 127.1.210.126:41363 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.210.66@KRBTEST.COM" created.
I20260806 02:46:01.985082 8088 ts_manager.cc:194] Registered new tserver with Master: cea96d710f5c430db2f412a731bd1bee (127.1.210.65:42461)
I20260806 02:46:01.987398 8088 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.210.65@KRBTEST.COM'} at 127.1.210.65:42951
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.66.keytab.
Entry for principal oryx/127.1.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.66.keytab.
I20260806 02:46:02.016392 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.66:0
--local_ip_for_outbound_sockets=127.1.210.66
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:41363
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.66.keytab
--principal=oryx/127.1.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:42477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:46:02.358990 8285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:02.359280 8285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:02.359437 8285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:02.367269 8285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:02.367394 8285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:02.367565 8285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.66
I20260806 02:46:02.377671 8285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:42477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.66.keytab
--principal=oryx/127.1.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:41363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.66
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:46:02.379104 8285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:02.380501 8285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:02.394371 8290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:02.394340 8291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:02.394599 8293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:02.395340 8285 server_base.cc:1034] running on GCE node
I20260806 02:46:02.395920 8285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:02.396835 8285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:02.398221 8285 hybrid_clock.cc:648] HybridClock initialized: now 1785984362398160 us; error 52 us; skew 500 ppm
I20260806 02:46:02.398599 8285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:46:02 dist-test-slave-8tdl krb5kdc[8042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984362, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:02.401989 8285 init.cc:377] Logged in from keytab as oryx/127.1.210.66@KRBTEST.COM (short username oryx)
I20260806 02:46:02.403650 8285 webserver.cc:460] Webserver started at http://127.1.210.66:37203/ using document root <none> and password file <none>
I20260806 02:46:02.404419 8285 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:02.404570 8285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:02.404894 8285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:02.408133 8285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-1/data/instance:
uuid: "f4874064dc7841d2b5578c350facdba9"
format_stamp: "Formatted at 2026-08-06 02:46:02 on dist-test-slave-8tdl"
server_key: "7149dcea356b91f9abc5babe9f83ac3d"
server_key_iv: "0b3230e768bcde7f91262263dfac56e5"
server_key_version: "encryptionkey@0"
I20260806 02:46:02.408990 8285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance:
uuid: "f4874064dc7841d2b5578c350facdba9"
format_stamp: "Formatted at 2026-08-06 02:46:02 on dist-test-slave-8tdl"
server_key: "7149dcea356b91f9abc5babe9f83ac3d"
server_key_iv: "0b3230e768bcde7f91262263dfac56e5"
server_key_version: "encryptionkey@0"
I20260806 02:46:02.414808 8285 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260806 02:46:02.418728 8300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:02.420445 8285 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 02:46:02.420646 8285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-1/wal
uuid: "f4874064dc7841d2b5578c350facdba9"
format_stamp: "Formatted at 2026-08-06 02:46:02 on dist-test-slave-8tdl"
server_key: "7149dcea356b91f9abc5babe9f83ac3d"
server_key_iv: "0b3230e768bcde7f91262263dfac56e5"
server_key_version: "encryptionkey@0"
I20260806 02:46:02.420874 8285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:02.437731 8285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:02.442446 8285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:02.442838 8285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:02.443922 8285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:46:02.445432 8285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:02.445537 8285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:02.445700 8285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:02.445775 8285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:02.481354 8285 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.66:34719
I20260806 02:46:02.481451 8413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.66:34719 every 8 connection(s)
I20260806 02:46:02.482848 8285 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
I20260806 02:46:02.491645 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 8285
I20260806 02:46:02.491878 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance
I20260806 02:46:02.492365 1865 external_mini_cluster.cc:1428] Setting key 5b63f6c01f41bbd381ef9094b5a98617
Aug 06 02:46:02 dist-test-slave-8tdl krb5kdc[8042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984362, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM
I20260806 02:46:02.506461 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:02.485811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:36375 (local address 127.1.210.126:41363)
0806 02:46:02.486363 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:02.486368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:02.487339 (+ 971us) server_negotiation.cc:407] Connection header received
0806 02:46:02.489464 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:02.489471 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:02.489592 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:02.489729 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:02.491566 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:02.492501 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:02.495029 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:02.495501 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:02.495654 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:02.499562 (+ 3908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:02.499567 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:02.499570 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:02.499607 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:02.503178 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:02.503988 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:02.503994 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:02.504004 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:02.504149 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:02.504637 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:02.504644 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:02.504652 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:02.504939 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:02.505108 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:02.505829 (+ 721us) server_negotiation.cc:299] Negotiation successful
0806 02:46:02.506072 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":326,"threads_started":1}
I20260806 02:46:02.507526 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:02.486278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.1.210.66:36375)
0806 02:46:02.487146 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:02.487197 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:02.489141 (+ 1944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:02.490013 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:02.490023 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:02.490629 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:02.491359 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:02.491378 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:02.492688 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:02.492692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:02.494820 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:02.494836 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:02.495757 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:02.495767 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:02.496048 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:02.496812 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:02.496838 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:02.499278 (+ 2440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:02.503328 (+ 4050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:02.503338 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:02.503356 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:02.503834 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:02.504272 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:02.504277 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:02.504280 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:02.504506 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:02.505082 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:02.505091 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:02.505460 (+ 369us) client_negotiation.cc:769] Sending connection context
0806 02:46:02.505737 (+ 277us) client_negotiation.cc:240] Negotiation successful
0806 02:46:02.506290 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":379,"threads_started":1}
I20260806 02:46:02.509261 8414 heartbeater.cc:347] Connected to a master server at 127.1.210.126:41363
I20260806 02:46:02.509599 8414 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:02.510360 8414 heartbeater.cc:511] Master 127.1.210.126:41363 requested a full tablet report, sending...
I20260806 02:46:02.512065 8088 ts_manager.cc:194] Registered new tserver with Master: f4874064dc7841d2b5578c350facdba9 (127.1.210.66:34719)
I20260806 02:46:02.513430 8088 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.210.66@KRBTEST.COM'} at 127.1.210.66:36375
WARNING: no policy specified for oryx/127.1.210.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.210.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.67.keytab.
Entry for principal oryx/127.1.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.67.keytab.
I20260806 02:46:02.550889 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.67:0
--local_ip_for_outbound_sockets=127.1.210.67
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:41363
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.67.keytab
--principal=oryx/127.1.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:42477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:46:02.892349 8420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:02.892661 8420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:02.892832 8420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:02.900560 8420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:02.900693 8420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:02.900887 8420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.67
I20260806 02:46:02.911197 8420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:42477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.67.keytab
--principal=oryx/127.1.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:41363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.67
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:46:02.912595 8420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:02.913962 8420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:02.928220 8426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:02.928154 8429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:02.928216 8427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:02.929611 8420 server_base.cc:1034] running on GCE node
I20260806 02:46:02.930272 8420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:02.931478 8420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:02.932730 8420 hybrid_clock.cc:648] HybridClock initialized: now 1785984362932679 us; error 98 us; skew 500 ppm
I20260806 02:46:02.933070 8420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:46:02 dist-test-slave-8tdl krb5kdc[8042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984362, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:02.939579 8420 init.cc:377] Logged in from keytab as oryx/127.1.210.67@KRBTEST.COM (short username oryx)
I20260806 02:46:02.941310 8420 webserver.cc:460] Webserver started at http://127.1.210.67:37767/ using document root <none> and password file <none>
I20260806 02:46:02.942149 8420 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:02.942284 8420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:02.942616 8420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:02.946007 8420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-2/data/instance:
uuid: "04982070d1fd4a2db220580b0b57043c"
format_stamp: "Formatted at 2026-08-06 02:46:02 on dist-test-slave-8tdl"
server_key: "942fd16a549e595f4d3c5a226c271032"
server_key_iv: "5b153645d14e074cf04e06ef1385533c"
server_key_version: "encryptionkey@0"
I20260806 02:46:02.946934 8420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance:
uuid: "04982070d1fd4a2db220580b0b57043c"
format_stamp: "Formatted at 2026-08-06 02:46:02 on dist-test-slave-8tdl"
server_key: "942fd16a549e595f4d3c5a226c271032"
server_key_iv: "5b153645d14e074cf04e06ef1385533c"
server_key_version: "encryptionkey@0"
I20260806 02:46:02.952745 8420 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 02:46:02.957185 8436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:02.959030 8420 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:46:02.959245 8420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-2/wal
uuid: "04982070d1fd4a2db220580b0b57043c"
format_stamp: "Formatted at 2026-08-06 02:46:02 on dist-test-slave-8tdl"
server_key: "942fd16a549e595f4d3c5a226c271032"
server_key_iv: "5b153645d14e074cf04e06ef1385533c"
server_key_version: "encryptionkey@0"
I20260806 02:46:02.959487 8420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:02.975090 8420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:02.980013 8420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:02.980337 8420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:02.981429 8420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:46:02.982887 8420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:02.983001 8420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:02.983157 8420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:02.983235 8420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:02.991201 8279 heartbeater.cc:503] Master 127.1.210.126:41363 was elected leader, sending a full tablet report...
I20260806 02:46:03.019060 8420 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.67:33381
I20260806 02:46:03.019083 8550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.67:33381 every 8 connection(s)
I20260806 02:46:03.020660 8420 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
I20260806 02:46:03.025540 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 8420
I20260806 02:46:03.025794 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance
I20260806 02:46:03.026394 1865 external_mini_cluster.cc:1428] Setting key be05fb407eb4737567167008460d3a18
Aug 06 02:46:03 dist-test-slave-8tdl krb5kdc[8042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984362, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM
I20260806 02:46:03.044289 8553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:03.023662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:55595 (local address 127.1.210.126:41363)
0806 02:46:03.024202 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:03.024207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:03.025331 (+ 1124us) server_negotiation.cc:407] Connection header received
0806 02:46:03.027883 (+ 2552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:03.027891 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:03.027992 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:03.028113 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:03.030237 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.031198 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.032960 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.033390 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.033495 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:03.037222 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:03.037227 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:03.037231 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:03.037269 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:03.040336 (+ 3067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:03.041445 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:03.041451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:03.041454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:03.041623 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:03.042145 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:03.042149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:03.042152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:03.042564 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:03.042754 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:03.043576 (+ 822us) server_negotiation.cc:299] Negotiation successful
0806 02:46:03.043823 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":276,"threads_started":1}
I20260806 02:46:03.045356 8554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:03.024273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.1.210.67:55595)
0806 02:46:03.025114 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:03.025176 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:03.027518 (+ 2342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:03.028347 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:03.028356 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:03.029168 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:03.030042 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:03.030064 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.031347 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.031351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:03.032784 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:03.032797 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.033592 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.033597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:03.033800 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:03.034449 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:03.034471 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:03.036932 (+ 2461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:03.040579 (+ 3647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:03.040591 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:03.040612 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:03.041253 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:03.041755 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:03.041760 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:03.041764 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:03.041980 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:03.042730 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:03.042739 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:03.043157 (+ 418us) client_negotiation.cc:769] Sending connection context
0806 02:46:03.043467 (+ 310us) client_negotiation.cc:240] Negotiation successful
0806 02:46:03.044066 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":283,"threads_started":1}
I20260806 02:46:03.046942 8551 heartbeater.cc:347] Connected to a master server at 127.1.210.126:41363
I20260806 02:46:03.047358 8551 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:03.048168 8551 heartbeater.cc:511] Master 127.1.210.126:41363 requested a full tablet report, sending...
I20260806 02:46:03.049856 8088 ts_manager.cc:194] Registered new tserver with Master: 04982070d1fd4a2db220580b0b57043c (127.1.210.67:33381)
I20260806 02:46:03.050734 8088 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.210.67@KRBTEST.COM'} at 127.1.210.67:55595
I20260806 02:46:03.055591 1865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.210.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.210.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.210.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.125.keytab.
Entry for principal oryx/127.1.210.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.125.keytab.
I20260806 02:46:03.112478 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.125:36719
--webserver_interface=127.1.210.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.125.keytab
--principal=oryx/127.1.210.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:42477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:41363,127.1.210.125:36719
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:46:03.463068 8557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:03.463435 8557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:03.463536 8557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:03.471300 8557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:46:03.471424 8557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:03.471542 8557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:03.471647 8557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:46:03.471725 8557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:46:03.482339 8557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:42477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:41363,127.1.210.125:36719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.125.keytab
--principal=oryx/127.1.210.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.125:36719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.210.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
W20260806 02:46:03.482815 8557 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.1.210.126:41363,127.1.210.125:36719'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260806 02:46:03.483920 8557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:03.485357 8557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:03.498291 8562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:03.498466 8563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:03.498440 8565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:03.499536 8557 server_base.cc:1034] running on GCE node
I20260806 02:46:03.500433 8557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:03.502038 8557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:03.503307 8557 hybrid_clock.cc:648] HybridClock initialized: now 1785984363503238 us; error 47 us; skew 500 ppm
I20260806 02:46:03.503700 8557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:46:03 dist-test-slave-8tdl krb5kdc[8042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984363, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:03.507195 8557 init.cc:377] Logged in from keytab as oryx/127.1.210.125@KRBTEST.COM (short username oryx)
I20260806 02:46:03.508816 8557 webserver.cc:460] Webserver started at http://127.1.210.125:46267/ using document root <none> and password file <none>
I20260806 02:46:03.509538 8557 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:03.509632 8557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:03.509898 8557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:03.513126 8557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-1/data/instance:
uuid: "88068f45c0904cde8fc22a8e54bc94c0"
format_stamp: "Formatted at 2026-08-06 02:46:03 on dist-test-slave-8tdl"
server_key: "93b2a76641d0072b70c0110ee9546a7f"
server_key_iv: "078123c9e80fcf388ea8258c27c2dacf"
server_key_version: "encryptionkey@0"
I20260806 02:46:03.513916 8557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-1/wal/instance:
uuid: "88068f45c0904cde8fc22a8e54bc94c0"
format_stamp: "Formatted at 2026-08-06 02:46:03 on dist-test-slave-8tdl"
server_key: "93b2a76641d0072b70c0110ee9546a7f"
server_key_iv: "078123c9e80fcf388ea8258c27c2dacf"
server_key_version: "encryptionkey@0"
I20260806 02:46:03.516539 8414 heartbeater.cc:503] Master 127.1.210.126:41363 was elected leader, sending a full tablet report...
I20260806 02:46:03.519892 8557 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260806 02:46:03.523805 8572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:03.525468 8557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:46:03.525691 8557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-1/wal
uuid: "88068f45c0904cde8fc22a8e54bc94c0"
format_stamp: "Formatted at 2026-08-06 02:46:03 on dist-test-slave-8tdl"
server_key: "93b2a76641d0072b70c0110ee9546a7f"
server_key_iv: "078123c9e80fcf388ea8258c27c2dacf"
server_key_version: "encryptionkey@0"
I20260806 02:46:03.525921 8557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:03.568352 8557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:03.573706 8557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:03.573976 8557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:03.596498 8557 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.125:36719
I20260806 02:46:03.596516 8623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.125:36719 every 8 connection(s)
I20260806 02:46:03.597987 8557 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-1/data/info.pb
I20260806 02:46:03.598618 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 8557
I20260806 02:46:03.603328 8624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:03.610666 8624 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:46:03 dist-test-slave-8tdl krb5kdc[8042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.210.125@KRBTEST.COM
Aug 06 02:46:03 dist-test-slave-8tdl krb5kdc[8042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984363, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.125@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM
I20260806 02:46:03.623988 8627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:03.601374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34262 (local address 127.1.210.125:36719)
0806 02:46:03.602340 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:03.602355 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:03.602408 (+ 53us) server_negotiation.cc:407] Connection header received
0806 02:46:03.604342 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:03.604401 (+ 59us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:03.604950 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:03.605420 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:03.607133 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.608567 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.609924 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.610657 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.610892 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:03.616413 (+ 5521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:03.616422 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:03.616432 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:03.616473 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:03.619609 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:03.620349 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:03.620356 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:03.620362 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:03.620540 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:03.620947 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:03.620953 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:03.620958 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:03.621552 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:03.621870 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:03.622377 (+ 507us) server_negotiation.cc:299] Negotiation successful
0806 02:46:03.622916 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"spinlock_wait_cycles":24192,"thread_start_us":442,"threads_started":1}
I20260806 02:46:03.629670 8628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:03.614417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56380 (local address 127.1.210.126:41363)
0806 02:46:03.615039 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:03.615048 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:03.615633 (+ 585us) server_negotiation.cc:407] Connection header received
0806 02:46:03.615886 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:03.615890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:03.615974 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:03.616097 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:03.617370 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.618140 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.619687 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.620101 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.620237 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:03.623566 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:03.623571 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:03.623574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:03.623613 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:03.626580 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:03.627518 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:03.627522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:03.627524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:03.627648 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:03.628040 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:03.628045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:03.628049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:03.628384 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:03.628556 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:03.629035 (+ 479us) server_negotiation.cc:299] Negotiation successful
0806 02:46:03.629271 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":321,"threads_started":1}
I20260806 02:46:03.629691 8629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:03.614408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.0.0.1:56380)
0806 02:46:03.615489 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:03.615527 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:03.615731 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:03.616178 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:03.616185 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:03.616899 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:03.617212 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:03.617221 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.618284 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.618289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:03.619538 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:03.619550 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.620302 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.620310 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:03.620482 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:03.621097 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:03.621120 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:03.623362 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:03.626730 (+ 3368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:03.626744 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:03.626750 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:03.627341 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:03.627762 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:03.627767 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:03.627771 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:03.627921 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:03.628524 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:03.628532 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:03.628705 (+ 173us) client_negotiation.cc:769] Sending connection context
0806 02:46:03.628905 (+ 200us) client_negotiation.cc:240] Negotiation successful
0806 02:46:03.629264 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"spinlock_wait_cycles":896,"thread_start_us":651,"threads_started":1}
I20260806 02:46:03.632028 8624 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:46:03 dist-test-slave-8tdl krb5kdc[8042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984363, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.125@KRBTEST.COM for oryx/127.1.210.125@KRBTEST.COM
I20260806 02:46:03.646948 8629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:03.632758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:36719 (local address 127.0.0.1:34270)
0806 02:46:03.632983 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:03.633000 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:03.633090 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:03.633340 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:03.633342 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:03.633668 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:03.633803 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:03.633810 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.635470 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.635473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:03.636309 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:03.636319 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.637200 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.637206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:03.637371 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:03.638078 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:03.638098 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:03.640639 (+ 2541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:03.644297 (+ 3658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:03.644301 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:03.644304 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:03.644696 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:03.645232 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:03.645236 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:03.645238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:03.645376 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:03.646046 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:03.646049 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:03.646177 (+ 128us) client_negotiation.cc:769] Sending connection context
0806 02:46:03.646296 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 02:46:03.646563 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260806 02:46:03.647284 8627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:03.632761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34270 (local address 127.1.210.125:36719)
0806 02:46:03.632993 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:03.632997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:03.633057 (+ 60us) server_negotiation.cc:407] Connection header received
0806 02:46:03.633174 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:03.633177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:03.633228 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:03.633303 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:03.634072 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.635330 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.636455 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.636871 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.637041 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:03.640836 (+ 3795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:03.640842 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:03.640845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:03.640876 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:03.644116 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:03.644874 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:03.644880 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:03.644883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:03.645086 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:03.645501 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:03.645505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:03.645507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:03.645846 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:03.646013 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:03.646441 (+ 428us) server_negotiation.cc:299] Negotiation successful
0806 02:46:03.646791 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":53}
I20260806 02:46:03.652263 8624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0: Bootstrap starting.
I20260806 02:46:03.655618 8624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:03.656716 8624 log.cc:824] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0: Log is configured to *not* fsync() on all Append() calls
I20260806 02:46:03.659780 8624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0: No bootstrap required, opened a new log
I20260806 02:46:03.667220 8624 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } }
I20260806 02:46:03.667557 8624 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:03.667692 8624 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88068f45c0904cde8fc22a8e54bc94c0, State: Initialized, Role: FOLLOWER
I20260806 02:46:03.668326 8624 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } }
I20260806 02:46:03.669240 8631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } } }
I20260806 02:46:03.669545 8631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:46:03.670348 8624 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 02:46:03.681401 8642 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:46:03.681586 8642 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:46:03.684084 8557 master_runner.cc:428] Detected that this master 88068f45c0904cde8fc22a8e54bc94c0 is joining an existing cluster
I20260806 02:46:03.684212 8557 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.210.125:36719
WARNING: no policy specified for oryx/127.1.210.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.210.124@KRBTEST.COM" created.
I20260806 02:46:03.724356 8627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:03.688846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34274 (local address 127.1.210.125:36719)
0806 02:46:03.689031 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:03.689040 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:03.704876 (+ 15836us) server_negotiation.cc:407] Connection header received
0806 02:46:03.705155 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:03.705163 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:03.705246 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:03.705325 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:03.707564 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.708786 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.711388 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.712216 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.712336 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:03.715902 (+ 3566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:03.715912 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:03.715917 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:03.715954 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:03.720550 (+ 4596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:03.721699 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:03.721704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:03.721707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:03.721881 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:03.722375 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:03.722380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:03.722383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:03.722809 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:03.722971 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:03.723677 (+ 706us) server_negotiation.cc:299] Negotiation successful
0806 02:46:03.723939 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 02:46:03.724340 8628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:03.688596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56396 (local address 127.1.210.126:41363)
0806 02:46:03.688815 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:03.688823 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:03.704149 (+ 15326us) server_negotiation.cc:407] Connection header received
0806 02:46:03.705259 (+ 1110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:03.705265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:03.705397 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:03.705480 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:03.707644 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.708702 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.711414 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.712231 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.712335 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:03.715932 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:03.715939 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:03.715944 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:03.715991 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:03.720522 (+ 4531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:03.721721 (+ 1199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:03.721728 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:03.721731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:03.721907 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:03.722390 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:03.722396 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:03.722399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:03.722816 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:03.722971 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:03.723696 (+ 725us) server_negotiation.cc:299] Negotiation successful
0806 02:46:03.723952 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 02:46:03.725575 8649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:03.688549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.0.0.1:56396)
0806 02:46:03.703943 (+ 15394us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:03.703980 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:03.705076 (+ 1096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:03.705720 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:03.705727 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:03.706630 (+ 903us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:03.707401 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:03.707419 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.708877 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.708885 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:03.711198 (+ 2313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:03.711219 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.712709 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.712721 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:03.712961 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:03.714009 (+ 1048us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:03.714040 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:03.715679 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:03.720724 (+ 5045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:03.720730 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:03.720734 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:03.721553 (+ 819us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:03.722054 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:03.722059 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:03.722062 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:03.722248 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:03.723049 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:03.723054 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:03.723381 (+ 327us) client_negotiation.cc:769] Sending connection context
0806 02:46:03.724640 (+ 1259us) client_negotiation.cc:240] Negotiation successful
0806 02:46:03.724968 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15132,"mutex_wait_us":282,"spinlock_wait_cycles":12928,"thread_start_us":16253,"threads_started":1}
I20260806 02:46:03.732131 8650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:03.689311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:36719 (local address 127.0.0.1:34274)
0806 02:46:03.704772 (+ 15461us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:03.704792 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:03.705007 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:03.705547 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:03.705556 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:03.706630 (+ 1074us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:03.707313 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:03.707332 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.708992 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.708997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:03.711198 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:03.711219 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.712736 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.712743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:03.712960 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:03.714060 (+ 1100us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:03.714078 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:03.715679 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:03.720724 (+ 5045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:03.720730 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:03.720734 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:03.721553 (+ 819us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:03.722041 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:03.722046 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:03.722050 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:03.722248 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:03.723049 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:03.723054 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:03.723370 (+ 316us) client_negotiation.cc:769] Sending connection context
0806 02:46:03.725094 (+ 1724us) client_negotiation.cc:240] Negotiation successful
0806 02:46:03.725405 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15316,"mutex_wait_us":341,"spinlock_wait_cycles":22784,"thread_start_us":15097,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.210.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.124.keytab.
Entry for principal oryx/127.1.210.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.124.keytab.
Aug 06 02:46:03 dist-test-slave-8tdl krb5kdc[8042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984361, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.126@KRBTEST.COM for oryx/127.1.210.125@KRBTEST.COM
I20260806 02:46:03.755378 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.124:44901
--webserver_interface=127.1.210.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.124.keytab
--principal=oryx/127.1.210.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:42477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:41363,127.1.210.125:36719,127.1.210.124:44901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:46:03.764165 8627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:03.737796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34286 (local address 127.1.210.125:36719)
0806 02:46:03.738012 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:03.738018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:03.744075 (+ 6057us) server_negotiation.cc:407] Connection header received
0806 02:46:03.744566 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:03.744571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:03.744673 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:03.744770 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:03.746567 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.747580 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.749823 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.750431 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.750554 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:03.755470 (+ 4916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:03.755478 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:03.755485 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:03.755537 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:03.759971 (+ 4434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:03.761145 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:03.761153 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:03.761156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:03.761379 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:03.761882 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:03.761890 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:03.761896 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:03.762406 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:03.762734 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:03.763315 (+ 581us) server_negotiation.cc:299] Negotiation successful
0806 02:46:03.763612 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 02:46:03.764194 8652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:03.737207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:36719 (local address 127.0.0.1:34286)
0806 02:46:03.743863 (+ 6656us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:03.743926 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:03.744242 (+ 316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:03.744893 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:03.744901 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:03.745939 (+ 1038us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:03.746277 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:03.746291 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.747745 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.747751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:03.749557 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:03.749577 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.750631 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.750638 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:03.750888 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:03.751827 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:03.751848 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:03.755165 (+ 3317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:03.760173 (+ 5008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:03.760194 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:03.760203 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:03.760943 (+ 740us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:03.761544 (+ 601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:03.761550 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:03.761554 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:03.761743 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:03.762710 (+ 967us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:03.762721 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:03.762943 (+ 222us) client_negotiation.cc:769] Sending connection context
0806 02:46:03.763189 (+ 246us) client_negotiation.cc:240] Negotiation successful
0806 02:46:03.763713 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6303,"thread_start_us":6128,"threads_started":1}
I20260806 02:46:03.767230 8088 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.210.125:36719: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0"
member_type: NON_VOTER
last_known_addr {
host: "127.1.210.125"
port: 36719
}
attrs {
promote: true
}
}
dest_uuid: "869bf3f6f6ad40e5ba34f73d3004a761"
cas_config_opid_index: -1
I20260806 02:46:03.768858 8088 consensus_queue.cc:237] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: NON_VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: true } }
I20260806 02:46:03.781231 8599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:03.782245 8655 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 88068f45c0904cde8fc22a8e54bc94c0 (127.1.210.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: NON_VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: true } } }
I20260806 02:46:03.785879 8599 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [term 1 FOLLOWER]: Refusing update from remote peer 869bf3f6f6ad40e5ba34f73d3004a761: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260806 02:46:03.797209 8657 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: NON_VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:46:03.801091 8655 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.210.125:36719
I20260806 02:46:03.803906 8557 master.cc:537] Master@127.1.210.125:36719 shutting down...
I20260806 02:46:03.807686 8656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: NON_VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: true } } }
I20260806 02:46:03.807988 8656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [sys.catalog]: This master's current role is: LEADER
I20260806 02:46:03.839097 8599 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 88068f45c0904cde8fc22a8e54bc94c0 (127.1.210.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: NON_VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: true } } }
I20260806 02:46:03.847287 8666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: NON_VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: true } } }
I20260806 02:46:03.847787 8666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [sys.catalog]: This master's current role is: LEARNER
I20260806 02:46:03.847761 8631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 869bf3f6f6ad40e5ba34f73d3004a761. Latest consensus state: current_term: 1 leader_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: NON_VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: true } } }
I20260806 02:46:03.848042 8631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [sys.catalog]: This master's current role is: LEARNER
I20260806 02:46:03.848366 8662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: NON_VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: true } } }
I20260806 02:46:03.848723 8662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [sys.catalog]: This master's current role is: LEARNER
I20260806 02:46:03.852052 8557 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [term 1 LEARNER]: Raft consensus shutting down.
I20260806 02:46:03.852567 8557 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [term 1 LEARNER]: Raft consensus is shut down!
I20260806 02:46:03.852738 8557 tablet_replica.cc:330] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0: stopping tablet replica
I20260806 02:46:03.871706 8557 master.cc:552] Master@127.1.210.125:36719 shutdown complete.
I20260806 02:46:03.871869 8557 master_runner.cc:305] Clearing existing system tablet
I20260806 02:46:03.872354 8557 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:03.872483 8557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:03.875304 8557 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20260806 02:46:03.876880 8668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:03.877246 8557 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:03.877552 8557 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260806 02:46:03.877645 8557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-1/wal
uuid: "88068f45c0904cde8fc22a8e54bc94c0"
format_stamp: "Formatted at 2026-08-06 02:46:03 on dist-test-slave-8tdl"
server_key: "93b2a76641d0072b70c0110ee9546a7f"
server_key_iv: "078123c9e80fcf388ea8258c27c2dacf"
server_key_version: "encryptionkey@0"
I20260806 02:46:03.880910 8557 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:46:03.886341 8557 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 02:46:03.886555 8557 log.cc:1196] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260806 02:46:03.887090 8557 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0: Deleting consensus metadata
I20260806 02:46:03.887372 8557 master_runner.cc:315] Copying system tablet from 127.1.210.126:41363
I20260806 02:46:03.895081 8557 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0: tablet copy: Beginning tablet copy session from remote peer at address 127.1.210.126:41363
I20260806 02:46:03.911598 8628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:03.896218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56406 (local address 127.1.210.126:41363)
0806 02:46:03.896463 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:03.896474 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:03.899594 (+ 3120us) server_negotiation.cc:407] Connection header received
0806 02:46:03.899846 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:03.899851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:03.899934 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:03.900059 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:03.900926 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.901665 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.903125 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.903612 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.903752 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:03.905803 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:03.905808 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:03.905811 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:03.905847 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:03.908662 (+ 2815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:03.909394 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:03.909402 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:03.909410 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:03.909543 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:03.909953 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:03.909959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:03.909963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:03.910260 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:03.910426 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:03.910926 (+ 500us) server_negotiation.cc:299] Negotiation successful
0806 02:46:03.911176 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 02:46:03.911655 8674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:03.896191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.0.0.1:56406)
0806 02:46:03.899457 (+ 3266us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:03.899480 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:03.899679 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:03.900090 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:03.900094 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:03.900546 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:03.900766 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:03.900776 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.901832 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.901838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:03.902969 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:03.902983 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.903771 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.903777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:03.903984 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:03.904690 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:03.904710 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:03.905564 (+ 854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:03.908829 (+ 3265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:03.908835 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:03.908839 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:03.909244 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:03.909662 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:03.909668 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:03.909671 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:03.909837 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:03.910386 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:03.910391 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:03.910567 (+ 176us) client_negotiation.cc:769] Sending connection context
0806 02:46:03.910813 (+ 246us) client_negotiation.cc:240] Negotiation successful
0806 02:46:03.911184 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3117,"thread_start_us":2972,"threads_started":1}
I20260806 02:46:03.912932 8109 tablet_copy_service.cc:140] P 869bf3f6f6ad40e5ba34f73d3004a761: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 88068f45c0904cde8fc22a8e54bc94c0 ({username='oryx', principal='oryx/127.1.210.125@KRBTEST.COM'} at 127.0.0.1:56406)
I20260806 02:46:03.913407 8109 tablet_copy_service.cc:161] P 869bf3f6f6ad40e5ba34f73d3004a761: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 88068f45c0904cde8fc22a8e54bc94c0 at {username='oryx', principal='oryx/127.1.210.125@KRBTEST.COM'} at 127.0.0.1:56406: session id = 88068f45c0904cde8fc22a8e54bc94c0-00000000000000000000000000000000
I20260806 02:46:03.916922 8109 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 02:46:03.919309 8557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 02:46:03.924403 8557 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0: tablet copy: Starting download of 0 data blocks...
I20260806 02:46:03.924765 8557 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0: tablet copy: Starting download of 1 WAL segments...
I20260806 02:46:03.928385 8557 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 02:46:03.932833 8109 tablet_copy_service.cc:342] P 869bf3f6f6ad40e5ba34f73d3004a761: Request end of tablet copy session 88068f45c0904cde8fc22a8e54bc94c0-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.210.125@KRBTEST.COM'} at 127.0.0.1:56406
I20260806 02:46:03.933178 8109 tablet_copy_service.cc:433] P 869bf3f6f6ad40e5ba34f73d3004a761: ending tablet copy session 88068f45c0904cde8fc22a8e54bc94c0-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 88068f45c0904cde8fc22a8e54bc94c0
I20260806 02:46:03.944909 8557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:03.947793 8675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:03.952566 8676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:03.968557 8678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:03.968875 8557 server_base.cc:1034] running on GCE node
I20260806 02:46:03.969566 8557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:03.970296 8557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:03.971547 8557 hybrid_clock.cc:648] HybridClock initialized: now 1785984363971457 us; error 63 us; skew 500 ppm
I20260806 02:46:03.971818 8557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:46:03 dist-test-slave-8tdl krb5kdc[8042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984363, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:03.973843 8557 init.cc:377] Logged in from keytab as oryx/127.1.210.125@KRBTEST.COM (short username oryx)
I20260806 02:46:03.978967 8557 webserver.cc:460] Webserver started at http://127.1.210.125:44919/ using document root <none> and password file <none>
I20260806 02:46:03.979446 8557 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:03.979554 8557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:03.981981 8557 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:46:03.983844 8685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:03.984326 8557 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 02:46:03.984513 8557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-1/wal
uuid: "88068f45c0904cde8fc22a8e54bc94c0"
format_stamp: "Formatted at 2026-08-06 02:46:03 on dist-test-slave-8tdl"
server_key: "93b2a76641d0072b70c0110ee9546a7f"
server_key_iv: "078123c9e80fcf388ea8258c27c2dacf"
server_key_version: "encryptionkey@0"
I20260806 02:46:03.984686 8557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:04.010596 8557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:04.015174 8557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:04.053727 8551 heartbeater.cc:503] Master 127.1.210.126:41363 was elected leader, sending a full tablet report...
I20260806 02:46:04.057600 8557 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.125:36719
I20260806 02:46:04.057610 8736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.125:36719 every 8 connection(s)
I20260806 02:46:04.058584 8557 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-1/data/info.pb
I20260806 02:46:04.066892 8737 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:46:04.069595 8737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0: Bootstrap starting.
I20260806 02:46:04.081676 8737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:46:04.082449 8737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0: Bootstrap complete.
I20260806 02:46:04.083899 8737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: NON_VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: true } }
I20260806 02:46:04.084210 8737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 88068f45c0904cde8fc22a8e54bc94c0, State: Initialized, Role: LEARNER
I20260806 02:46:04.084713 8737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: NON_VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: true } }
I20260806 02:46:04.086154 8740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: NON_VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: true } } }
I20260806 02:46:04.086642 8737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:46:04.086721 8740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [sys.catalog]: This master's current role is: LEARNER
I20260806 02:46:04.109086 8751 catalog_manager.cc:1259] Loaded cluster ID: fa59af352fb5431abc8784c5d36b6618
I20260806 02:46:04.109230 8751 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0: loading cluster ID for follower catalog manager: success
I20260806 02:46:04.112085 8751 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0: acquiring CA information for follower catalog manager: success
I20260806 02:46:04.116098 8751 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 02:46:04.319957 8653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:04.320264 8653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:04.320392 8653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 02:46:04.328548 8656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: NON_VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: true } } }
I20260806 02:46:04.328909 8656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [sys.catalog]: This master's current role is: LEADER
W20260806 02:46:04.336467 8653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:46:04.336735 8653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:04.336905 8653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:04.337118 8653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:46:04.337275 8653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:46:04.350893 8758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:04.329391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:36719 (local address 127.0.0.1:45050)
0806 02:46:04.336942 (+ 7551us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:04.336966 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:04.337282 (+ 316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:04.338294 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:04.338299 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:04.338341 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:46:04.338660 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:04.338672 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.342904 (+ 4232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.342910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:04.346087 (+ 3177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:04.346120 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.349610 (+ 3490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.349617 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:04.349921 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:04.349998 (+ 77us) client_negotiation.cc:769] Sending connection context
0806 02:46:04.350107 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 02:46:04.350254 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7349,"thread_start_us":7171,"threads_started":1}
I20260806 02:46:04.351084 8759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:04.329891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45050 (local address 127.1.210.125:36719)
0806 02:46:04.336085 (+ 6194us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:04.336091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:04.337463 (+ 1372us) server_negotiation.cc:407] Connection header received
0806 02:46:04.337546 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:04.337553 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:04.337641 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:04.337769 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:46:04.338977 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.342565 (+ 3588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.346352 (+ 3787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.349322 (+ 2970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.349511 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:04.349951 (+ 440us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:04.350173 (+ 222us) server_negotiation.cc:299] Negotiation successful
0806 02:46:04.350375 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5926,"thread_start_us":5754,"threads_started":1}
I20260806 02:46:04.352850 8740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 869bf3f6f6ad40e5ba34f73d3004a761. Latest consensus state: current_term: 1 leader_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: NON_VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: true } } }
I20260806 02:46:04.353291 8740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [sys.catalog]: This master's current role is: LEARNER
I20260806 02:46:04.354960 8656 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761: attempting to promote NON_VOTER 88068f45c0904cde8fc22a8e54bc94c0 to VOTER
I20260806 02:46:04.356734 8653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:42477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:41363,127.1.210.125:36719,127.1.210.124:44901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.124.keytab
--principal=oryx/127.1.210.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.124:44901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.210.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:46:04.356865 8656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false } }
I20260806 02:46:04.358951 8653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:04.359853 8712 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [term 1 LEARNER]: Refusing update from remote peer 869bf3f6f6ad40e5ba34f73d3004a761: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 02:46:04.361019 8653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:46:04.360889 8656 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 02:46:04.367647 8756 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 88068f45c0904cde8fc22a8e54bc94c0 (127.1.210.125) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false } } }
I20260806 02:46:04.371775 8712 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 88068f45c0904cde8fc22a8e54bc94c0 (127.1.210.125) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false } } }
I20260806 02:46:04.376097 8755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false } } }
I20260806 02:46:04.377002 8755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [sys.catalog]: This master's current role is: LEADER
I20260806 02:46:04.377611 8755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false } } }
I20260806 02:46:04.378059 8755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [sys.catalog]: This master's current role is: LEADER
I20260806 02:46:04.379127 8740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false } } }
I20260806 02:46:04.379437 8740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [sys.catalog]: This master's current role is: FOLLOWER
W20260806 02:46:04.394912 8763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:04.395042 8762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:04.395138 8765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:04.396796 8653 server_base.cc:1034] running on GCE node
I20260806 02:46:04.397816 8653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:04.399443 8653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:04.400710 8653 hybrid_clock.cc:648] HybridClock initialized: now 1785984364400601 us; error 44 us; skew 500 ppm
I20260806 02:46:04.401141 8653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:46:04 dist-test-slave-8tdl krb5kdc[8042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984364, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:04.405860 8653 init.cc:377] Logged in from keytab as oryx/127.1.210.124@KRBTEST.COM (short username oryx)
I20260806 02:46:04.408038 8653 webserver.cc:460] Webserver started at http://127.1.210.124:34983/ using document root <none> and password file <none>
I20260806 02:46:04.408951 8653 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:04.409173 8653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:04.409590 8653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:04.414003 8653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-2/data/instance:
uuid: "9cd0128d2476470cb51e77201cfb727a"
format_stamp: "Formatted at 2026-08-06 02:46:04 on dist-test-slave-8tdl"
server_key: "dcc5bc7c8cd73ecdd21322b9da5b39bb"
server_key_iv: "82e3ce1746a30f33a6330f30fa26a464"
server_key_version: "encryptionkey@0"
I20260806 02:46:04.415138 8653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-2/wal/instance:
uuid: "9cd0128d2476470cb51e77201cfb727a"
format_stamp: "Formatted at 2026-08-06 02:46:04 on dist-test-slave-8tdl"
server_key: "dcc5bc7c8cd73ecdd21322b9da5b39bb"
server_key_iv: "82e3ce1746a30f33a6330f30fa26a464"
server_key_version: "encryptionkey@0"
I20260806 02:46:04.421968 8653 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260806 02:46:04.427191 8772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:04.428776 8653 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:46:04.429041 8653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-2/wal
uuid: "9cd0128d2476470cb51e77201cfb727a"
format_stamp: "Formatted at 2026-08-06 02:46:04 on dist-test-slave-8tdl"
server_key: "dcc5bc7c8cd73ecdd21322b9da5b39bb"
server_key_iv: "82e3ce1746a30f33a6330f30fa26a464"
server_key_version: "encryptionkey@0"
I20260806 02:46:04.429356 8653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:04.460654 8653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:04.465929 8653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:04.466252 8653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:04.510835 8653 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.124:44901
I20260806 02:46:04.511247 8823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.124:44901 every 8 connection(s)
I20260806 02:46:04.512406 8653 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-2/data/info.pb
I20260806 02:46:04.518172 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:04.522917 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 8653
I20260806 02:46:04.530272 8824 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:46:04 dist-test-slave-8tdl krb5kdc[8042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.210.124@KRBTEST.COM
Aug 06 02:46:04 dist-test-slave-8tdl krb5kdc[8042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984364, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.124@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM
I20260806 02:46:04.561690 8829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:04.534139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41404 (local address 127.1.210.126:41363)
0806 02:46:04.535650 (+ 1511us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:04.535657 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:04.535826 (+ 169us) server_negotiation.cc:407] Connection header received
0806 02:46:04.536111 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:04.536117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:04.536234 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:04.536474 (+ 240us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:04.539264 (+ 2790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.540905 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.544285 (+ 3380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.545278 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.545397 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:04.551843 (+ 6446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:04.551852 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:04.551859 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:04.551920 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:04.557004 (+ 5084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:04.558513 (+ 1509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:04.558519 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:04.558522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:04.558852 (+ 330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:04.559478 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:04.559484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:04.559488 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:04.560000 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:04.560125 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:04.560909 (+ 784us) server_negotiation.cc:299] Negotiation successful
0806 02:46:04.561192 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1371,"thread_start_us":1090,"threads_started":1}
I20260806 02:46:04.562323 8828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:04.533904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.0.0.1:41404)
0806 02:46:04.535591 (+ 1687us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:04.535649 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:04.535961 (+ 312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:04.536596 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:04.536614 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:04.537861 (+ 1247us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:04.538930 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:04.538946 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.541139 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.541143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:04.544057 (+ 2914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:04.544076 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.545717 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.545729 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:04.546070 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:04.547276 (+ 1206us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:04.547351 (+ 75us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:04.551541 (+ 4190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:04.557222 (+ 5681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:04.557234 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:04.557243 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:04.558188 (+ 945us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:04.558993 (+ 805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:04.558996 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:04.558999 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:04.559278 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:04.560138 (+ 860us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:04.560145 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:04.560421 (+ 276us) client_negotiation.cc:769] Sending connection context
0806 02:46:04.560721 (+ 300us) client_negotiation.cc:240] Negotiation successful
0806 02:46:04.561134 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1090,"spinlock_wait_cycles":27008,"thread_start_us":563,"threads_started":1}
I20260806 02:46:04.562390 8827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:04.526045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49970 (local address 127.1.210.124:44901)
0806 02:46:04.527669 (+ 1624us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:04.527682 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:04.527746 (+ 64us) server_negotiation.cc:407] Connection header received
0806 02:46:04.530373 (+ 2627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:04.530399 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:04.531009 (+ 610us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:04.531573 (+ 564us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:04.534545 (+ 2972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.537643 (+ 3098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.540024 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.541483 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.541724 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:04.550634 (+ 8910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:04.550650 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:04.550713 (+ 63us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:04.550765 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:04.555020 (+ 4255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:04.556303 (+ 1283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:04.556311 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:04.556320 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:04.556571 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:04.557221 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:04.557227 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:04.557231 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:04.558305 (+ 1074us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:04.558568 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:04.559508 (+ 940us) server_negotiation.cc:299] Negotiation successful
0806 02:46:04.560078 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1358,"thread_start_us":984,"threads_started":1}
I20260806 02:46:04.565369 8824 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:46:04 dist-test-slave-8tdl krb5kdc[8042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984364, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.124@KRBTEST.COM for oryx/127.1.210.125@KRBTEST.COM
I20260806 02:46:04.583691 8828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:04.566811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:36719 (local address 127.0.0.1:45062)
0806 02:46:04.567080 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:04.567097 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:04.567195 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:04.567584 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:04.567588 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:04.568068 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:04.568318 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:04.568327 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.569857 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.569861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:04.571421 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:04.571435 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.572525 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.572530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:04.572814 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:04.573697 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:04.573719 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:04.577036 (+ 3317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:04.580730 (+ 3694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:04.580735 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:04.580739 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:04.581291 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:04.581764 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:04.581769 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:04.581773 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:04.581933 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:04.582495 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:04.582503 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:04.582737 (+ 234us) client_negotiation.cc:769] Sending connection context
0806 02:46:04.582890 (+ 153us) client_negotiation.cc:240] Negotiation successful
0806 02:46:04.583251 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260806 02:46:04.583716 8759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:04.566549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45062 (local address 127.1.210.125:36719)
0806 02:46:04.566825 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:04.566831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:04.567175 (+ 344us) server_negotiation.cc:407] Connection header received
0806 02:46:04.567374 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:04.567379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:04.567443 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:04.567540 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:04.568587 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.569697 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.571592 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.572370 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.572491 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:04.577331 (+ 4840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:04.577338 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:04.577342 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:04.577380 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:04.580570 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:04.581474 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:04.581485 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:04.581487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:04.581635 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:04.582060 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:04.582072 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:04.582074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:04.582364 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:04.582601 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:04.583005 (+ 404us) server_negotiation.cc:299] Negotiation successful
0806 02:46:04.583266 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260806 02:46:04.585222 8824 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44901 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:46:04 dist-test-slave-8tdl krb5kdc[8042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984364, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.124@KRBTEST.COM for oryx/127.1.210.124@KRBTEST.COM
I20260806 02:46:04.601436 8827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:04.586313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49978 (local address 127.1.210.124:44901)
0806 02:46:04.586515 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:04.586520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:04.587128 (+ 608us) server_negotiation.cc:407] Connection header received
0806 02:46:04.587188 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:04.587192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:04.587255 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:04.587345 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:04.588291 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.589214 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.590539 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.591182 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.591329 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:04.595365 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:04.595371 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:04.595374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:04.595403 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:04.598643 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:04.599453 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:04.599458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:04.599462 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:04.599625 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:04.600005 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:04.600013 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:04.600016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:04.600291 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:04.600449 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:04.600775 (+ 326us) server_negotiation.cc:299] Negotiation successful
0806 02:46:04.601116 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":145,"server-negotiator.queue_time_us":84}
I20260806 02:46:04.601531 8828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:04.586727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:44901 (local address 127.0.0.1:49978)
0806 02:46:04.586899 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:04.586918 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:04.587046 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:04.587383 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:04.587387 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:04.587957 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:04.588139 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:04.588156 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.589343 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.589351 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:04.590386 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:04.590398 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.591338 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.591343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:04.591633 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:04.592393 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:04.592412 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:04.595155 (+ 2743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:04.598875 (+ 3720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:04.598882 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:04.598886 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:04.599319 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:04.599736 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:04.599742 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:04.599745 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:04.599894 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:04.600404 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:04.600408 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:04.600543 (+ 135us) client_negotiation.cc:769] Sending connection context
0806 02:46:04.600695 (+ 152us) client_negotiation.cc:240] Negotiation successful
0806 02:46:04.600985 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 02:46:04.608386 8824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a: Bootstrap starting.
I20260806 02:46:04.613276 8824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:04.615137 8824 log.cc:824] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a: Log is configured to *not* fsync() on all Append() calls
I20260806 02:46:04.619483 8824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a: No bootstrap required, opened a new log
I20260806 02:46:04.630577 8824 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } } peers { permanent_uuid: "9cd0128d2476470cb51e77201cfb727a" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44901 } }
I20260806 02:46:04.631182 8824 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:04.631417 8824 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cd0128d2476470cb51e77201cfb727a, State: Initialized, Role: FOLLOWER
I20260806 02:46:04.632077 8824 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } } peers { permanent_uuid: "9cd0128d2476470cb51e77201cfb727a" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44901 } }
I20260806 02:46:04.633319 8831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } } peers { permanent_uuid: "9cd0128d2476470cb51e77201cfb727a" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44901 } } }
I20260806 02:46:04.634157 8831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:46:04.634896 8824 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [sys.catalog]: configured and running, proceeding with master startup.
W20260806 02:46:04.652885 8842 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:46:04.653185 8842 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:46:04.656833 8653 master_runner.cc:428] Detected that this master 9cd0128d2476470cb51e77201cfb727a is joining an existing cluster
I20260806 02:46:04.657038 8653 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.210.124:44901
I20260806 02:46:04.670583 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 8150
I20260806 02:46:04.699744 8829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:04.662246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41418 (local address 127.1.210.126:41363)
0806 02:46:04.662479 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:04.662486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:04.663708 (+ 1222us) server_negotiation.cc:407] Connection header received
0806 02:46:04.664143 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:04.664151 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:04.664313 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:04.664436 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:04.666954 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.668594 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.679246 (+ 10652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.680110 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.680259 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:04.688540 (+ 8281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:04.688560 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:04.688574 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:04.688647 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:04.694708 (+ 6061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:04.695869 (+ 1161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:04.695875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:04.695879 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:04.696209 (+ 330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:04.697113 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:04.697119 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:04.697123 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:04.697701 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:04.698138 (+ 437us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:04.698705 (+ 567us) server_negotiation.cc:299] Negotiation successful
0806 02:46:04.699174 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 02:46:04.700301 8848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:04.661772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.0.0.1:41418)
0806 02:46:04.663440 (+ 1668us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:04.663461 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:04.663895 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:04.664795 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:04.664803 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:04.665918 (+ 1115us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:04.666710 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:04.666730 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.668881 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.668886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:04.678532 (+ 9646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:04.678549 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.683352 (+ 4803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.683362 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:04.683798 (+ 436us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:04.684866 (+ 1068us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:04.684964 (+ 98us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:04.688241 (+ 3277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:04.694894 (+ 6653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:04.694900 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:04.694904 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:04.695634 (+ 730us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:04.696409 (+ 775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:04.696414 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:04.696417 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:04.696753 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:04.697862 (+ 1109us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:04.697867 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:04.698295 (+ 428us) client_negotiation.cc:769] Sending connection context
0806 02:46:04.698506 (+ 211us) client_negotiation.cc:240] Negotiation successful
0806 02:46:04.699063 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1527,"thread_start_us":1375,"threads_started":1}
Aug 06 02:46:04 dist-test-slave-8tdl krb5kdc[8042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.210.65@KRBTEST.COM
I20260806 02:46:04.716102 8759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:04.661756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45064 (local address 127.1.210.125:36719)
0806 02:46:04.661958 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:04.661965 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:04.680958 (+ 18993us) server_negotiation.cc:407] Connection header received
0806 02:46:04.681364 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:04.681370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:04.681471 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:04.681545 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:04.684747 (+ 3202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.686443 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.694612 (+ 8169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.695913 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.696067 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:04.704614 (+ 8547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:04.704622 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:04.704626 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:04.704672 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:04.710493 (+ 5821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:04.712550 (+ 2057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:04.712560 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:04.712566 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:04.712786 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:04.713406 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:04.713415 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:04.713421 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:04.713889 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:04.714078 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:04.714912 (+ 834us) server_negotiation.cc:299] Negotiation successful
0806 02:46:04.715354 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 02:46:04.716368 8850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:04.662000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:44901 (local address 127.0.0.1:49992)
0806 02:46:04.676359 (+ 14359us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:04.676389 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:04.676664 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:04.677420 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:04.677428 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:04.679438 (+ 2010us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:04.680745 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:04.680772 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.685820 (+ 5048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.685839 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:04.689373 (+ 3534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:04.689393 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.693761 (+ 4368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.693769 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:04.694048 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:04.697163 (+ 3115us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:04.697190 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:04.700670 (+ 3480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:04.712931 (+ 12261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:04.712936 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:04.712939 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:04.713604 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:04.714102 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:04.714108 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:04.714111 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:04.714304 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:04.715198 (+ 894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:04.715203 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:04.715363 (+ 160us) client_negotiation.cc:769] Sending connection context
0806 02:46:04.715505 (+ 142us) client_negotiation.cc:240] Negotiation successful
0806 02:46:04.715826 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14065,"thread_start_us":18095,"threads_started":1}
I20260806 02:46:04.717288 8827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:04.662337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49992 (local address 127.1.210.124:44901)
0806 02:46:04.662624 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:04.662632 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:04.676499 (+ 13867us) server_negotiation.cc:407] Connection header received
0806 02:46:04.676825 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:04.676834 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:04.676940 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:04.677024 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:04.680929 (+ 3905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.682916 (+ 1987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.691603 (+ 8687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.692885 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.693066 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:04.705478 (+ 12412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:04.705485 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:04.705490 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:04.705527 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:04.712766 (+ 7239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:04.713742 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:04.713749 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:04.713752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:04.713971 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:04.714451 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:04.714457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:04.714461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:04.714971 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:04.715144 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:04.716694 (+ 1550us) server_negotiation.cc:299] Negotiation successful
0806 02:46:04.716912 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:46:04.718636 8849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:04.663897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:36719 (local address 127.0.0.1:45064)
0806 02:46:04.680779 (+ 16882us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:04.680805 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:04.681142 (+ 337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:04.681847 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:04.681852 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:04.683260 (+ 1408us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:04.684107 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:04.684131 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.686753 (+ 2622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.686758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:04.691354 (+ 4596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:04.691370 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.698849 (+ 7479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.698860 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:04.699186 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:04.702159 (+ 2973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:04.702198 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:04.704307 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:04.710743 (+ 6436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:04.710752 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:04.710759 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:04.712341 (+ 1582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:04.712992 (+ 651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:04.713000 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:04.713014 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:04.713248 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:04.714181 (+ 933us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:04.714188 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:04.714466 (+ 278us) client_negotiation.cc:769] Sending connection context
0806 02:46:04.716918 (+ 2452us) client_negotiation.cc:240] Negotiation successful
0806 02:46:04.717770 (+ 852us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16565,"thread_start_us":11447,"threads_started":1}
I20260806 02:46:04.719316 8851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:04.672464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52188 (local address 127.1.210.65:42461)
0806 02:46:04.681456 (+ 8992us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:04.681467 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:04.681498 (+ 31us) server_negotiation.cc:407] Connection header received
0806 02:46:04.681607 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:04.681618 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:04.682060 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:04.685671 (+ 3611us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:04.686197 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.688869 (+ 2672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.691629 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.693080 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.703319 (+ 10239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:04.703957 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:04.703977 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:04.704010 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:04.704065 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:04.714352 (+ 10287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:04.715561 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:04.715571 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:04.715574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:04.715741 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:04.716265 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:04.716272 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:04.716278 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:04.716898 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:04.718395 (+ 1497us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:04.718562 (+ 167us) server_negotiation.cc:299] Negotiation successful
0806 02:46:04.718874 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8754,"thread_start_us":10581,"threads_started":1}
W20260806 02:46:04.719605 8778 tablet.cc:2369] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:46:04.721099 8274 generic_service.cc:196] Checking for leaks (request via RPC)
Aug 06 02:46:04 dist-test-slave-8tdl krb5kdc[8042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984361, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.126@KRBTEST.COM for oryx/127.1.210.124@KRBTEST.COM
I20260806 02:46:04.743633 8758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:04.708757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:44901 (local address 127.0.0.1:50000)
0806 02:46:04.709218 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:04.709238 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:04.709372 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:04.717992 (+ 8620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:04.717999 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:04.718929 (+ 930us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:04.719286 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:04.719305 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.722670 (+ 3365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.722713 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:04.723721 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:04.723737 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.725268 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.725314 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:04.725606 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:04.727070 (+ 1464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:04.727108 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:04.731722 (+ 4614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:04.738404 (+ 6682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:04.738413 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:04.738419 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:04.739506 (+ 1087us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:04.740394 (+ 888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:04.740402 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:04.740406 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:04.740735 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:04.741924 (+ 1189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:04.741930 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:04.742309 (+ 379us) client_negotiation.cc:769] Sending connection context
0806 02:46:04.742509 (+ 200us) client_negotiation.cc:240] Negotiation successful
0806 02:46:04.742963 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335}
I20260806 02:46:04.745504 8827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:04.709713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50000 (local address 127.1.210.124:44901)
0806 02:46:04.717530 (+ 7817us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:04.717535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:04.717551 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:46:04.717645 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:04.717651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:04.717755 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:04.717858 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:04.721041 (+ 3183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.722500 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.723886 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.725006 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.728652 (+ 3646us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:04.731986 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:04.731992 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:04.731998 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:04.732037 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:04.738131 (+ 6094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:04.739731 (+ 1600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:04.739738 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:04.739744 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:04.740149 (+ 405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:04.740933 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:04.741012 (+ 79us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:04.741017 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:04.741729 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:04.744162 (+ 2433us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:04.744366 (+ 204us) server_negotiation.cc:299] Negotiation successful
0806 02:46:04.744869 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7702,"thread_start_us":21894,"threads_started":1}
I20260806 02:46:04.747582 8088 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.210.124:44901: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9cd0128d2476470cb51e77201cfb727a"
member_type: NON_VOTER
last_known_addr {
host: "127.1.210.124"
port: 44901
}
attrs {
promote: true
}
}
dest_uuid: "869bf3f6f6ad40e5ba34f73d3004a761"
cas_config_opid_index: 6
I20260806 02:46:04.748581 8088 consensus_queue.cc:237] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false } } peers { permanent_uuid: "9cd0128d2476470cb51e77201cfb727a" member_type: NON_VOTER last_known_addr { host: "127.1.210.124" port: 44901 } attrs { promote: true } }
I20260806 02:46:04.751768 8712 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [term 1 FOLLOWER]: Refusing update from remote peer 869bf3f6f6ad40e5ba34f73d3004a761: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260806 02:46:04.751955 8799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:04.752722 8755 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 02:46:04.756670 8756 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9cd0128d2476470cb51e77201cfb727a (127.1.210.124) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false } } peers { permanent_uuid: "9cd0128d2476470cb51e77201cfb727a" member_type: NON_VOTER last_known_addr { host: "127.1.210.124" port: 44901 } attrs { promote: true } } }
I20260806 02:46:04.758200 8799 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [term 1 FOLLOWER]: Refusing update from remote peer 869bf3f6f6ad40e5ba34f73d3004a761: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260806 02:46:04.759569 8755 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cd0128d2476470cb51e77201cfb727a" member_type: NON_VOTER last_known_addr { host: "127.1.210.124" port: 44901 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:46:04.761736 8712 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9cd0128d2476470cb51e77201cfb727a (127.1.210.124) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false } } peers { permanent_uuid: "9cd0128d2476470cb51e77201cfb727a" member_type: NON_VOTER last_known_addr { host: "127.1.210.124" port: 44901 } attrs { promote: true } } }
I20260806 02:46:04.765589 8756 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.210.124:44901
I20260806 02:46:04.766559 8653 master.cc:537] Master@127.1.210.124:44901 shutting down...
I20260806 02:46:04.767549 8755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false } } peers { permanent_uuid: "9cd0128d2476470cb51e77201cfb727a" member_type: NON_VOTER last_known_addr { host: "127.1.210.124" port: 44901 } attrs { promote: true } } }
I20260806 02:46:04.767867 8755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [sys.catalog]: This master's current role is: LEADER
I20260806 02:46:04.768169 8656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false } } peers { permanent_uuid: "9cd0128d2476470cb51e77201cfb727a" member_type: NON_VOTER last_known_addr { host: "127.1.210.124" port: 44901 } attrs { promote: true } } }
I20260806 02:46:04.768683 8656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [sys.catalog]: This master's current role is: LEADER
I20260806 02:46:04.768822 8740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false } } peers { permanent_uuid: "9cd0128d2476470cb51e77201cfb727a" member_type: NON_VOTER last_known_addr { host: "127.1.210.124" port: 44901 } attrs { promote: true } } }
I20260806 02:46:04.769100 8740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [sys.catalog]: This master's current role is: FOLLOWER
W20260806 02:46:04.772603 8072 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260806 02:46:04.784185 8072 consensus_peers.cc:487] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 -> Peer 9cd0128d2476470cb51e77201cfb727a (127.1.210.124:44901): Couldn't send request to peer 9cd0128d2476470cb51e77201cfb727a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 02:46:04.810065 8653 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:46:04.810570 8653 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:46:04.810843 8653 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a: stopping tablet replica
I20260806 02:46:04.829432 8653 master.cc:552] Master@127.1.210.124:44901 shutdown complete.
I20260806 02:46:04.829576 8653 master_runner.cc:305] Clearing existing system tablet
I20260806 02:46:04.830041 8653 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:04.830166 8653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:04.833925 8653 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260806 02:46:04.836068 8856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:04.837188 8653 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:04.837651 8653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260806 02:46:04.837803 8653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-2/wal
uuid: "9cd0128d2476470cb51e77201cfb727a"
format_stamp: "Formatted at 2026-08-06 02:46:04 on dist-test-slave-8tdl"
server_key: "dcc5bc7c8cd73ecdd21322b9da5b39bb"
server_key_iv: "82e3ce1746a30f33a6330f30fa26a464"
server_key_version: "encryptionkey@0"
I20260806 02:46:04.843834 8653 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:46:04.851239 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 8150
I20260806 02:46:04.855103 8653 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 02:46:04.855311 8653 log.cc:1196] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260806 02:46:04.855783 8653 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a: Deleting consensus metadata
I20260806 02:46:04.856055 8653 master_runner.cc:315] Copying system tablet from 127.1.210.126:41363
I20260806 02:46:04.864131 8653 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a: tablet copy: Beginning tablet copy session from remote peer at address 127.1.210.126:41363
I20260806 02:46:04.876711 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.65:42461
--local_ip_for_outbound_sockets=127.1.210.65
--tserver_master_addrs=127.1.210.126:41363
--webserver_port=33851
--webserver_interface=127.1.210.65
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.65.keytab
--principal=oryx/127.1.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:42477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.210.126:41363,127.1.210.125:36719
--tserver_master_addrs=127.1.210.126:41363,127.1.210.125:36719,127.1.210.124:44901 with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:46:04.878659 8829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:04.865095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41422 (local address 127.1.210.126:41363)
0806 02:46:04.865418 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:04.865430 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:04.866005 (+ 575us) server_negotiation.cc:407] Connection header received
0806 02:46:04.866230 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:04.866236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:04.866333 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:04.866445 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:04.867387 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.868129 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.869525 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.869992 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.870130 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:04.872209 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:04.872214 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:04.872217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:04.872251 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:04.875263 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:04.876096 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:04.876102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:04.876105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:04.876273 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:04.876806 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:04.876812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:04.876816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:04.877249 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:04.877469 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:04.877992 (+ 523us) server_negotiation.cc:299] Negotiation successful
0806 02:46:04.878257 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260806 02:46:04.879185 8862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:04.865051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.0.0.1:41422)
0806 02:46:04.865890 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:04.865907 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:04.866084 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:04.866483 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:04.866488 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:04.867017 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:04.867232 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:04.867245 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.868304 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.868310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:04.869338 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:04.869352 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.870158 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.870165 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:04.870333 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:04.871176 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:04.871197 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:04.871967 (+ 770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:04.875395 (+ 3428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:04.875401 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:04.875405 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:04.875948 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:04.876429 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:04.876435 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:04.876438 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:04.876647 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:04.877401 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:04.877409 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:04.877675 (+ 266us) client_negotiation.cc:769] Sending connection context
0806 02:46:04.877854 (+ 179us) client_negotiation.cc:240] Negotiation successful
0806 02:46:04.878291 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":560,"threads_started":1}
I20260806 02:46:04.879920 8109 tablet_copy_service.cc:140] P 869bf3f6f6ad40e5ba34f73d3004a761: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9cd0128d2476470cb51e77201cfb727a ({username='oryx', principal='oryx/127.1.210.124@KRBTEST.COM'} at 127.0.0.1:41422)
I20260806 02:46:04.880116 8109 tablet_copy_service.cc:161] P 869bf3f6f6ad40e5ba34f73d3004a761: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9cd0128d2476470cb51e77201cfb727a at {username='oryx', principal='oryx/127.1.210.124@KRBTEST.COM'} at 127.0.0.1:41422: session id = 9cd0128d2476470cb51e77201cfb727a-00000000000000000000000000000000
I20260806 02:46:04.884338 8109 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 02:46:04.886410 8653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 02:46:04.891680 8653 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a: tablet copy: Starting download of 0 data blocks...
I20260806 02:46:04.891993 8653 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a: tablet copy: Starting download of 1 WAL segments...
I20260806 02:46:04.894403 8653 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 02:46:04.897768 8109 tablet_copy_service.cc:342] P 869bf3f6f6ad40e5ba34f73d3004a761: Request end of tablet copy session 9cd0128d2476470cb51e77201cfb727a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.210.124@KRBTEST.COM'} at 127.0.0.1:41422
I20260806 02:46:04.897913 8109 tablet_copy_service.cc:433] P 869bf3f6f6ad40e5ba34f73d3004a761: ending tablet copy session 9cd0128d2476470cb51e77201cfb727a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9cd0128d2476470cb51e77201cfb727a
I20260806 02:46:04.912853 8653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:04.915951 8864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:04.917541 8865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:04.924945 8653 server_base.cc:1034] running on GCE node
W20260806 02:46:04.924780 8867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:04.925570 8653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:04.926360 8653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:04.927662 8653 hybrid_clock.cc:648] HybridClock initialized: now 1785984364927606 us; error 37 us; skew 500 ppm
I20260806 02:46:04.927961 8653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:46:04 dist-test-slave-8tdl krb5kdc[8042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984364, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:04.930418 8653 init.cc:377] Logged in from keytab as oryx/127.1.210.124@KRBTEST.COM (short username oryx)
I20260806 02:46:04.935782 8653 webserver.cc:460] Webserver started at http://127.1.210.124:46735/ using document root <none> and password file <none>
I20260806 02:46:04.936185 8653 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:04.936324 8653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:04.939183 8653 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:46:04.941314 8874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:04.941869 8653 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260806 02:46:04.942014 8653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-2/wal
uuid: "9cd0128d2476470cb51e77201cfb727a"
format_stamp: "Formatted at 2026-08-06 02:46:04 on dist-test-slave-8tdl"
server_key: "dcc5bc7c8cd73ecdd21322b9da5b39bb"
server_key_iv: "82e3ce1746a30f33a6330f30fa26a464"
server_key_version: "encryptionkey@0"
I20260806 02:46:04.942238 8653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:04.951177 8653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:04.955519 8653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:04.991225 8653 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.124:44901
I20260806 02:46:04.991254 8925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.124:44901 every 8 connection(s)
I20260806 02:46:04.992215 8653 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/master-2/data/info.pb
I20260806 02:46:04.999898 8926 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:46:05.009176 8926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a: Bootstrap starting.
I20260806 02:46:05.028353 8926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:46:05.028880 8926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a: Bootstrap complete.
I20260806 02:46:05.029865 8926 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false } } peers { permanent_uuid: "9cd0128d2476470cb51e77201cfb727a" member_type: NON_VOTER last_known_addr { host: "127.1.210.124" port: 44901 } attrs { promote: true } }
I20260806 02:46:05.030334 8926 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9cd0128d2476470cb51e77201cfb727a, State: Initialized, Role: LEARNER
I20260806 02:46:05.030618 8926 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false } } peers { permanent_uuid: "9cd0128d2476470cb51e77201cfb727a" member_type: NON_VOTER last_known_addr { host: "127.1.210.124" port: 44901 } attrs { promote: true } }
I20260806 02:46:05.031693 8929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false } } peers { permanent_uuid: "9cd0128d2476470cb51e77201cfb727a" member_type: NON_VOTER last_known_addr { host: "127.1.210.124" port: 44901 } attrs { promote: true } } }
I20260806 02:46:05.032086 8926 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:46:05.032212 8929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [sys.catalog]: This master's current role is: LEARNER
I20260806 02:46:05.044793 8940 catalog_manager.cc:1259] Loaded cluster ID: fa59af352fb5431abc8784c5d36b6618
I20260806 02:46:05.044935 8940 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a: loading cluster ID for follower catalog manager: success
I20260806 02:46:05.048830 8940 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a: acquiring CA information for follower catalog manager: success
I20260806 02:46:05.051821 8940 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:46:05.209203 8758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:05.197383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:44901 (local address 127.0.0.1:50004)
0806 02:46:05.197764 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:05.197787 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:05.197957 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:05.199044 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:05.199049 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:05.199088 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:46:05.199299 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:05.199310 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:05.201866 (+ 2556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:05.201873 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:05.204171 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:05.204195 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:05.207736 (+ 3541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:05.207753 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:05.208053 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:05.208490 (+ 437us) client_negotiation.cc:769] Sending connection context
0806 02:46:05.208635 (+ 145us) client_negotiation.cc:240] Negotiation successful
0806 02:46:05.208842 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221}
I20260806 02:46:05.209810 8941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:05.197584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50004 (local address 127.1.210.124:44901)
0806 02:46:05.198556 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:05.198563 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:05.198584 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:46:05.198663 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:05.198669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:05.198804 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:05.198942 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:46:05.199540 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:05.201616 (+ 2076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:05.204454 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:05.206760 (+ 2306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:05.206963 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:05.207331 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:05.208707 (+ 1376us) server_negotiation.cc:299] Negotiation successful
0806 02:46:05.208921 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":610,"threads_started":1}
I20260806 02:46:05.212901 8901 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.7->[]
I20260806 02:46:05.214017 8929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 869bf3f6f6ad40e5ba34f73d3004a761. Latest consensus state: current_term: 1 leader_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false } } peers { permanent_uuid: "9cd0128d2476470cb51e77201cfb727a" member_type: NON_VOTER last_known_addr { host: "127.1.210.124" port: 44901 } attrs { promote: true } } }
I20260806 02:46:05.214766 8929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [sys.catalog]: This master's current role is: LEARNER
W20260806 02:46:05.371716 8863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:05.372035 8863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:05.372198 8863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:05.380801 8863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:05.380966 8863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:05.381156 8863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.65
I20260806 02:46:05.393402 8863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:42477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.65.keytab
--principal=oryx/127.1.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.65:42461
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.65
--webserver_port=33851
--tserver_master_addrs=127.1.210.126:41363,127.1.210.125:36719,127.1.210.124:44901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.65
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:46:05.395267 8863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:05.397360 8863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:05.417026 8946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:05.417171 8949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:05.417282 8947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:05.419158 8863 server_base.cc:1034] running on GCE node
I20260806 02:46:05.419912 8863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:05.421103 8863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:05.422416 8863 hybrid_clock.cc:648] HybridClock initialized: now 1785984365422335 us; error 70 us; skew 500 ppm
I20260806 02:46:05.423062 8863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:46:05 dist-test-slave-8tdl krb5kdc[8042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984365, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:05.428081 8863 init.cc:377] Logged in from keytab as oryx/127.1.210.65@KRBTEST.COM (short username oryx)
I20260806 02:46:05.430357 8863 webserver.cc:460] Webserver started at http://127.1.210.65:33851/ using document root <none> and password file <none>
I20260806 02:46:05.431595 8863 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:05.431878 8863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:05.439822 8863 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260806 02:46:05.444605 8956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:05.446596 8863 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.004s
I20260806 02:46:05.446874 8863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-0/wal
uuid: "cea96d710f5c430db2f412a731bd1bee"
format_stamp: "Formatted at 2026-08-06 02:46:01 on dist-test-slave-8tdl"
server_key: "7bb7675311034e5f40114255e889cd17"
server_key_iv: "f4a86902c1ef64f69b2da72bacdee46a"
server_key_version: "encryptionkey@0"
I20260806 02:46:05.448189 8863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:05.463828 8863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:05.470008 8863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:05.470485 8863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:05.471868 8863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:46:05.473977 8863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:05.474092 8863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:05.474265 8863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:05.474491 8863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:05.560216 8863 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.65:42461
I20260806 02:46:05.560328 9069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.65:42461 every 8 connection(s)
I20260806 02:46:05.562050 8863 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
I20260806 02:46:05.563141 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 8863
I20260806 02:46:05.563915 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 8285
Aug 06 02:46:05 dist-test-slave-8tdl krb5kdc[8042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.210.66@KRBTEST.COM
Aug 06 02:46:05 dist-test-slave-8tdl krb5kdc[8042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984365, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM
Aug 06 02:46:05 dist-test-slave-8tdl krb5kdc[8042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984365, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.124@KRBTEST.COM
Aug 06 02:46:05 dist-test-slave-8tdl krb5kdc[8042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984365, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.125@KRBTEST.COM
I20260806 02:46:05.620330 9073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:05.565111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51476 (local address 127.1.210.66:34719)
0806 02:46:05.566280 (+ 1169us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:05.566291 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:05.566342 (+ 51us) server_negotiation.cc:407] Connection header received
0806 02:46:05.566563 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:05.566576 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:05.566919 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:05.567124 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:05.569160 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:05.571341 (+ 2181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:05.574439 (+ 3098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:05.575818 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:05.575990 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:05.595072 (+ 19082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:05.595118 (+ 46us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:05.595139 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:05.595191 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:05.613688 (+ 18497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:05.615659 (+ 1971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:05.615670 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:05.615674 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:05.615946 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:05.616570 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:05.616575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:05.616578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:05.617170 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:05.619063 (+ 1893us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:05.619249 (+ 186us) server_negotiation.cc:299] Negotiation successful
0806 02:46:05.619548 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":930,"thread_start_us":680,"threads_started":1}
I20260806 02:46:05.621320 8941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:05.572611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:43059 (local address 127.1.210.124:44901)
0806 02:46:05.572878 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:05.572887 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:05.586716 (+ 13829us) server_negotiation.cc:407] Connection header received
0806 02:46:05.592685 (+ 5969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:05.592692 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:05.592788 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:05.592887 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:05.594450 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:05.595901 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:05.600634 (+ 4733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:05.601477 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:05.601589 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:05.610927 (+ 9338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:05.610947 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:05.610954 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:05.611004 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:05.616349 (+ 5345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:05.617592 (+ 1243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:05.617598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:05.617604 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:05.617840 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:05.618605 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:05.618613 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:05.618617 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:05.619086 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:05.619221 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:05.620536 (+ 1315us) server_negotiation.cc:299] Negotiation successful
0806 02:46:05.620808 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260806 02:46:05.622860 8409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:46:05.624351 9080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:05.571637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:44901 (local address 127.1.210.65:43059)
0806 02:46:05.586480 (+ 14843us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:05.586547 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:05.589825 (+ 3278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:05.593146 (+ 3321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:05.593150 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:05.593837 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:05.594277 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:05.594288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:05.596163 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:05.596169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:05.600408 (+ 4239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:05.600426 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:05.602108 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:05.602115 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:05.602557 (+ 442us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:05.603857 (+ 1300us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:05.603957 (+ 100us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:05.610498 (+ 6541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:05.616560 (+ 6062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:05.616572 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:05.616594 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:05.617412 (+ 818us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:05.618051 (+ 639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:05.618059 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:05.618063 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:05.618450 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:05.619318 (+ 868us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:05.619326 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:05.619887 (+ 561us) client_negotiation.cc:769] Sending connection context
0806 02:46:05.621533 (+ 1646us) client_negotiation.cc:240] Negotiation successful
0806 02:46:05.622236 (+ 703us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14306,"mutex_wait_us":99,"spinlock_wait_cycles":44288,"thread_start_us":8364,"threads_started":1}
I20260806 02:46:05.626744 9070 heartbeater.cc:347] Connected to a master server at 127.1.210.124:44901
I20260806 02:46:05.627285 9070 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:05.629532 9070 heartbeater.cc:511] Master 127.1.210.124:44901 requested a full tablet report, sending...
I20260806 02:46:05.631094 9076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:05.570243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:34831 (local address 127.1.210.126:41363)
0806 02:46:05.583887 (+ 13644us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:05.583896 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:05.587219 (+ 3323us) server_negotiation.cc:407] Connection header received
0806 02:46:05.590381 (+ 3162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:05.590388 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:05.590560 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:05.590674 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:05.594190 (+ 3516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:05.595912 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:05.600760 (+ 4848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:05.601895 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:05.602035 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:05.610586 (+ 8551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:05.610600 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:05.610609 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:05.610720 (+ 111us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:05.617500 (+ 6780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:05.626869 (+ 9369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:05.626876 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:05.626880 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:05.627162 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:05.628152 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:05.628158 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:05.628161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:05.628719 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:05.628952 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:05.629973 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0806 02:46:05.630269 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13465,"thread_start_us":13666,"threads_started":1}
I20260806 02:46:05.631789 9079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:05.583525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:49781 (local address 127.1.210.125:36719)
0806 02:46:05.594914 (+ 11389us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:05.594921 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:05.594960 (+ 39us) server_negotiation.cc:407] Connection header received
0806 02:46:05.595023 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:05.595032 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:05.595165 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:05.595278 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:05.598102 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:05.599782 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:05.602295 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:05.603065 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:05.603226 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:05.618571 (+ 15345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:05.618584 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:05.618593 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:05.618660 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:05.627421 (+ 8761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:05.628693 (+ 1272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:05.628700 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:05.628703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:05.629005 (+ 302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:05.629646 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:05.629651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:05.629655 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:05.630052 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:05.630239 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:05.631054 (+ 815us) server_negotiation.cc:299] Negotiation successful
0806 02:46:05.631324 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11219,"thread_start_us":10961,"threads_started":1}
I20260806 02:46:05.632683 9078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:05.583940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:36719 (local address 127.1.210.65:49781)
0806 02:46:05.590565 (+ 6625us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:05.591050 (+ 485us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:05.591175 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:05.595480 (+ 4305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:05.595485 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:05.597157 (+ 1672us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:05.597860 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:05.597877 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:05.600024 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:05.600031 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:05.602093 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:05.602107 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:05.603267 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:05.603274 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:05.603473 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:05.604243 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:05.604269 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:05.618169 (+ 13900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:05.627602 (+ 9433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:05.627611 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:05.627615 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:05.628529 (+ 914us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:05.629163 (+ 634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:05.629168 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:05.629172 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:05.629514 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:05.630322 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:05.630327 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:05.630549 (+ 222us) client_negotiation.cc:769] Sending connection context
0806 02:46:05.630774 (+ 225us) client_negotiation.cc:240] Negotiation successful
0806 02:46:05.632133 (+ 1359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6459,"mutex_wait_us":836,"thread_start_us":755,"threads_started":1}
I20260806 02:46:05.625167 8756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false } } peers { permanent_uuid: "9cd0128d2476470cb51e77201cfb727a" member_type: NON_VOTER last_known_addr { host: "127.1.210.124" port: 44901 } attrs { promote: true } } }
I20260806 02:46:05.633548 8756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [sys.catalog]: This master's current role is: LEADER
I20260806 02:46:05.633797 8891 ts_manager.cc:194] Registered new tserver with Master: cea96d710f5c430db2f412a731bd1bee (127.1.210.65:42461)
I20260806 02:46:05.633970 9077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:05.573285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.1.210.65:34831)
0806 02:46:05.587035 (+ 13750us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:05.587065 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:05.589825 (+ 2760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:05.591404 (+ 1579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:05.591417 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:05.592625 (+ 1208us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:05.593910 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:05.593947 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:05.596167 (+ 2220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:05.596173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:05.600445 (+ 4272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:05.600467 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:05.602446 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:05.602457 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:05.602771 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:05.603851 (+ 1080us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:05.603905 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:05.610050 (+ 6145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:05.617772 (+ 7722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:05.617779 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:05.617783 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:05.623873 (+ 6090us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:05.627628 (+ 3755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:05.627632 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:05.627635 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:05.627978 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:05.629045 (+ 1067us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:05.629052 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:05.629328 (+ 276us) client_negotiation.cc:769] Sending connection context
0806 02:46:05.629565 (+ 237us) client_negotiation.cc:240] Negotiation successful
0806 02:46:05.631645 (+ 2080us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13465,"mutex_wait_us":141,"thread_start_us":8389,"threads_started":1}
I20260806 02:46:05.634425 9074 heartbeater.cc:347] Connected to a master server at 127.1.210.125:36719
I20260806 02:46:05.634716 9074 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:05.635588 9071 heartbeater.cc:347] Connected to a master server at 127.1.210.126:41363
I20260806 02:46:05.635768 9071 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:05.635836 9074 heartbeater.cc:511] Master 127.1.210.125:36719 requested a full tablet report, sending...
I20260806 02:46:05.637027 9071 heartbeater.cc:511] Master 127.1.210.126:41363 requested a full tablet report, sending...
I20260806 02:46:05.637805 9083 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761: attempting to promote NON_VOTER 9cd0128d2476470cb51e77201cfb727a to VOTER
I20260806 02:46:05.638243 8702 ts_manager.cc:194] Registered new tserver with Master: cea96d710f5c430db2f412a731bd1bee (127.1.210.65:42461)
I20260806 02:46:05.638377 8088 ts_manager.cc:194] Re-registered known tserver with Master: cea96d710f5c430db2f412a731bd1bee (127.1.210.65:42461)
I20260806 02:46:05.639631 9083 consensus_queue.cc:237] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false } } peers { permanent_uuid: "9cd0128d2476470cb51e77201cfb727a" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44901 } attrs { promote: false } }
I20260806 02:46:05.640194 8088 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.210.65@KRBTEST.COM'} at 127.1.210.65:34831
I20260806 02:46:05.647811 8901 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [term 1 LEARNER]: Refusing update from remote peer 869bf3f6f6ad40e5ba34f73d3004a761: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260806 02:46:05.648190 8712 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [term 1 FOLLOWER]: Refusing update from remote peer 869bf3f6f6ad40e5ba34f73d3004a761: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260806 02:46:05.649935 9083 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cd0128d2476470cb51e77201cfb727a" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 02:46:05.650506 8756 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 02:46:05.655558 9083 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9cd0128d2476470cb51e77201cfb727a (127.1.210.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false } } peers { permanent_uuid: "9cd0128d2476470cb51e77201cfb727a" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44901 } attrs { promote: false } } }
I20260806 02:46:05.655921 8712 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9cd0128d2476470cb51e77201cfb727a (127.1.210.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false } } peers { permanent_uuid: "9cd0128d2476470cb51e77201cfb727a" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44901 } attrs { promote: false } } }
I20260806 02:46:05.663147 9084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false } } peers { permanent_uuid: "9cd0128d2476470cb51e77201cfb727a" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44901 } attrs { promote: false } } }
I20260806 02:46:05.664646 9084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [sys.catalog]: This master's current role is: LEADER
I20260806 02:46:05.664060 9083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false } } peers { permanent_uuid: "9cd0128d2476470cb51e77201cfb727a" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44901 } attrs { promote: false } } }
I20260806 02:46:05.665165 9083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [sys.catalog]: This master's current role is: LEADER
I20260806 02:46:05.667042 9083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false } } peers { permanent_uuid: "9cd0128d2476470cb51e77201cfb727a" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44901 } attrs { promote: false } } }
I20260806 02:46:05.667405 9083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761 [sys.catalog]: This master's current role is: LEADER
I20260806 02:46:05.666004 9088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false } } peers { permanent_uuid: "9cd0128d2476470cb51e77201cfb727a" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44901 } attrs { promote: false } } }
I20260806 02:46:05.667927 9088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:46:05.668355 8901 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9cd0128d2476470cb51e77201cfb727a (127.1.210.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false } } peers { permanent_uuid: "9cd0128d2476470cb51e77201cfb727a" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44901 } attrs { promote: false } } }
I20260806 02:46:05.672791 8929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false } } peers { permanent_uuid: "9cd0128d2476470cb51e77201cfb727a" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44901 } attrs { promote: false } } }
I20260806 02:46:05.673116 8929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:46:05.749861 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 8285
I20260806 02:46:05.767442 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.66:34719
--local_ip_for_outbound_sockets=127.1.210.66
--tserver_master_addrs=127.1.210.126:41363
--webserver_port=37203
--webserver_interface=127.1.210.66
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.66.keytab
--principal=oryx/127.1.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:42477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.210.126:41363,127.1.210.125:36719
--tserver_master_addrs=127.1.210.126:41363,127.1.210.125:36719,127.1.210.124:44901 with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:46:06.168423 9089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:06.168814 9089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:06.169006 9089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:06.177582 9089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:06.177754 9089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:06.177932 9089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.66
I20260806 02:46:06.189092 9089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:42477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.66.keytab
--principal=oryx/127.1.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.66:34719
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.66
--webserver_port=37203
--tserver_master_addrs=127.1.210.126:41363,127.1.210.125:36719,127.1.210.124:44901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.66
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:46:06.190615 9089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:06.192121 9089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:06.208331 9094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:06.208436 9097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:06.208539 9095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:06.210022 9089 server_base.cc:1034] running on GCE node
I20260806 02:46:06.210970 9089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:06.212069 9089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:06.213321 9089 hybrid_clock.cc:648] HybridClock initialized: now 1785984366213256 us; error 57 us; skew 500 ppm
I20260806 02:46:06.213721 9089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:46:06 dist-test-slave-8tdl krb5kdc[8042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984366, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:06.217408 9089 init.cc:377] Logged in from keytab as oryx/127.1.210.66@KRBTEST.COM (short username oryx)
I20260806 02:46:06.219195 9089 webserver.cc:460] Webserver started at http://127.1.210.66:37203/ using document root <none> and password file <none>
I20260806 02:46:06.219955 9089 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:06.220121 9089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:06.226421 9089 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:46:06.229521 9105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:06.231336 9089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:46:06.231580 9089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-1/wal
uuid: "f4874064dc7841d2b5578c350facdba9"
format_stamp: "Formatted at 2026-08-06 02:46:02 on dist-test-slave-8tdl"
server_key: "7149dcea356b91f9abc5babe9f83ac3d"
server_key_iv: "0b3230e768bcde7f91262263dfac56e5"
server_key_version: "encryptionkey@0"
I20260806 02:46:06.232719 9089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:06.248308 9089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:06.253188 9089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:06.253499 9089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:06.254566 9089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:46:06.256040 9089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:06.256174 9089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:06.256306 9089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:06.256394 9089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:06.294229 9089 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.66:34719
I20260806 02:46:06.294279 9218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.66:34719 every 8 connection(s)
I20260806 02:46:06.295759 9089 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
I20260806 02:46:06.299979 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 9089
I20260806 02:46:06.300307 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 8420
Aug 06 02:46:06 dist-test-slave-8tdl krb5kdc[8042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984366, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.125@KRBTEST.COM
Aug 06 02:46:06 dist-test-slave-8tdl krb5kdc[8042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984366, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM
I20260806 02:46:06.337108 9224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:06.299141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:54687 (local address 127.1.210.125:36719)
0806 02:46:06.300946 (+ 1805us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:06.300956 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:06.312246 (+ 11290us) server_negotiation.cc:407] Connection header received
0806 02:46:06.313348 (+ 1102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:06.313353 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:06.313441 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:06.313567 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:06.316606 (+ 3039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:06.317583 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:06.319869 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:06.320504 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:06.320635 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:06.327999 (+ 7364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:06.328007 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:06.328011 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:06.328057 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:06.332343 (+ 4286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:06.334096 (+ 1753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:06.334104 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:06.334108 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:06.334253 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:06.334783 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:06.334789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:06.334792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:06.335243 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:06.335378 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:06.336271 (+ 893us) server_negotiation.cc:299] Negotiation successful
0806 02:46:06.336580 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1663,"thread_start_us":3321,"threads_started":1}
I20260806 02:46:06.338511 9225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:06.302318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:36719 (local address 127.1.210.66:54687)
0806 02:46:06.312031 (+ 9713us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:06.312052 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:06.312865 (+ 813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:06.313897 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:06.313914 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:06.315230 (+ 1316us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:06.316332 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:06.316368 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:06.317767 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:06.317773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:06.319708 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:06.319720 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:06.320946 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:06.320953 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:06.322504 (+ 1551us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:06.323793 (+ 1289us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:06.323828 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:06.327693 (+ 3865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:06.332580 (+ 4887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:06.332584 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:06.332603 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:06.333947 (+ 1344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:06.334440 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:06.334447 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:06.334451 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:06.334613 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:06.335493 (+ 880us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:06.335497 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:06.335629 (+ 132us) client_negotiation.cc:769] Sending connection context
0806 02:46:06.335944 (+ 315us) client_negotiation.cc:240] Negotiation successful
0806 02:46:06.338075 (+ 2131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9572,"mutex_wait_us":1675,"spinlock_wait_cycles":1408,"thread_start_us":6938,"threads_started":1}
I20260806 02:46:06.339075 9223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:06.299090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:35577 (local address 127.1.210.126:41363)
0806 02:46:06.316487 (+ 17397us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:06.316496 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:06.316530 (+ 34us) server_negotiation.cc:407] Connection header received
0806 02:46:06.316597 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:06.316602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:06.316693 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:06.316806 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:06.319978 (+ 3172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:06.321044 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:06.322312 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:06.323008 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:06.323154 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:06.327999 (+ 4845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:06.328006 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:06.328011 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:06.328057 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:06.332383 (+ 4326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:06.333695 (+ 1312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:06.333702 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:06.333707 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:06.333899 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:06.334486 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:06.334492 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:06.334495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:06.334926 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:06.338218 (+ 3292us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:06.338390 (+ 172us) server_negotiation.cc:299] Negotiation successful
0806 02:46:06.338658 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17235,"thread_start_us":17818,"threads_started":1}
I20260806 02:46:06.338111 9230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:06.300813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.1.210.66:35577)
0806 02:46:06.311943 (+ 11130us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:06.311974 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:06.312865 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:06.317793 (+ 4928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:06.317800 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:06.318488 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:06.318813 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:06.318826 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:06.321167 (+ 2341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:06.321175 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:06.322153 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:06.322166 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:06.323210 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:06.323216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:06.323397 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:06.325064 (+ 1667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:06.325092 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:06.327717 (+ 2625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:06.332569 (+ 4852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:06.332582 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:06.332603 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:06.333511 (+ 908us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:06.334054 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:06.334061 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:06.334066 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:06.334335 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:06.335094 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:06.335103 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:06.335579 (+ 476us) client_negotiation.cc:769] Sending connection context
0806 02:46:06.335944 (+ 365us) client_negotiation.cc:240] Negotiation successful
0806 02:46:06.336568 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10920,"mutex_wait_us":3,"spinlock_wait_cycles":73984,"thread_start_us":10639,"threads_started":1}
I20260806 02:46:06.340577 9220 heartbeater.cc:347] Connected to a master server at 127.1.210.126:41363
I20260806 02:46:06.340653 9221 heartbeater.cc:347] Connected to a master server at 127.1.210.125:36719
I20260806 02:46:06.340962 9221 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:06.340970 9220 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:06.342084 9220 heartbeater.cc:511] Master 127.1.210.126:41363 requested a full tablet report, sending...
I20260806 02:46:06.342113 9221 heartbeater.cc:511] Master 127.1.210.125:36719 requested a full tablet report, sending...
I20260806 02:46:06.344115 8702 ts_manager.cc:194] Registered new tserver with Master: f4874064dc7841d2b5578c350facdba9 (127.1.210.66:34719)
Aug 06 02:46:06 dist-test-slave-8tdl krb5kdc[8042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.210.67@KRBTEST.COM
I20260806 02:46:06.344939 8088 ts_manager.cc:194] Re-registered known tserver with Master: f4874064dc7841d2b5578c350facdba9 (127.1.210.66:34719)
I20260806 02:46:06.345897 8088 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.210.66@KRBTEST.COM'} at 127.1.210.66:35577
Aug 06 02:46:06 dist-test-slave-8tdl krb5kdc[8042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984366, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.124@KRBTEST.COM
I20260806 02:46:06.355947 9226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:06.300888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:33767 (local address 127.1.210.124:44901)
0806 02:46:06.338963 (+ 38075us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:06.338967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:06.339022 (+ 55us) server_negotiation.cc:407] Connection header received
0806 02:46:06.339092 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:06.339097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:06.339179 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:06.340145 (+ 966us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:06.341360 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:06.342841 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:06.345245 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:06.345643 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:06.345783 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:06.349706 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:06.349712 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:06.349717 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:06.349758 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:06.353005 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:06.353736 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:06.353743 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:06.353747 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:06.353885 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:06.354339 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:06.354345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:06.354348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:06.354648 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:06.354934 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:06.355303 (+ 369us) server_negotiation.cc:299] Negotiation successful
0806 02:46:06.355550 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37955,"thread_start_us":37849,"threads_started":1}
I20260806 02:46:06.355985 9229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:06.299949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:44901 (local address 127.1.210.66:33767)
0806 02:46:06.309824 (+ 9875us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:06.309902 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:06.317034 (+ 7132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:06.340190 (+ 23156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:06.340195 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:06.340757 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:06.341027 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:06.341041 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:06.343008 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:06.343014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:06.345038 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:06.345052 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:06.345844 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:06.345852 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:06.346068 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:06.346761 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:06.346784 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:06.349508 (+ 2724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:06.353164 (+ 3656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:06.353170 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:06.353173 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:06.353614 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:06.354055 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:06.354060 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:06.354064 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:06.354224 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:06.354902 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:06.354907 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:06.355055 (+ 148us) client_negotiation.cc:769] Sending connection context
0806 02:46:06.355212 (+ 157us) client_negotiation.cc:240] Negotiation successful
0806 02:46:06.355550 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9434,"thread_start_us":927,"threads_started":1}
I20260806 02:46:06.356546 9228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:06.302232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36804 (local address 127.1.210.67:33381)
0806 02:46:06.325484 (+ 23252us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:06.325497 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:06.326741 (+ 1244us) server_negotiation.cc:407] Connection header received
0806 02:46:06.326846 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:06.326858 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:06.327351 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:06.327549 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:06.330835 (+ 3286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:06.333737 (+ 2902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:06.336969 (+ 3232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:06.338193 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:06.338361 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:06.347595 (+ 9234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:06.347610 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:06.347623 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:06.347659 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:06.352422 (+ 4763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:06.353597 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:06.353606 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:06.353609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:06.353768 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:06.354313 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:06.354318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:06.354320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:06.354795 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:06.355169 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:06.355546 (+ 377us) server_negotiation.cc:299] Negotiation successful
0806 02:46:06.355854 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22989,"thread_start_us":23266,"threads_started":1}
I20260806 02:46:06.357226 9219 heartbeater.cc:347] Connected to a master server at 127.1.210.124:44901
I20260806 02:46:06.357380 9219 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:06.357769 9219 heartbeater.cc:511] Master 127.1.210.124:44901 requested a full tablet report, sending...
I20260806 02:46:06.358296 8546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:46:06.359284 8891 ts_manager.cc:194] Registered new tserver with Master: f4874064dc7841d2b5578c350facdba9 (127.1.210.66:34719)
I20260806 02:46:06.445765 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 8420
I20260806 02:46:06.465003 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.67:33381
--local_ip_for_outbound_sockets=127.1.210.67
--tserver_master_addrs=127.1.210.126:41363
--webserver_port=37767
--webserver_interface=127.1.210.67
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.67.keytab
--principal=oryx/127.1.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:42477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.210.126:41363,127.1.210.125:36719
--tserver_master_addrs=127.1.210.126:41363,127.1.210.125:36719,127.1.210.124:44901 with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:46:06.644176 9071 heartbeater.cc:503] Master 127.1.210.126:41363 was elected leader, sending a full tablet report...
W20260806 02:46:06.822757 9232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:06.823104 9232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:06.823273 9232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:06.830930 9232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:06.831089 9232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:06.831310 9232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.67
I20260806 02:46:06.842309 9232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:42477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.67.keytab
--principal=oryx/127.1.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.67:33381
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.67
--webserver_port=37767
--tserver_master_addrs=127.1.210.126:41363,127.1.210.125:36719,127.1.210.124:44901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.67
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:46:06.844409 9232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:06.846381 9232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:06.861256 9237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:06.861223 9238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:06.861182 9240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:06.861861 9232 server_base.cc:1034] running on GCE node
I20260806 02:46:06.862648 9232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:06.863759 9232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:06.865051 9232 hybrid_clock.cc:648] HybridClock initialized: now 1785984366864992 us; error 50 us; skew 500 ppm
I20260806 02:46:06.865438 9232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:46:06 dist-test-slave-8tdl krb5kdc[8042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984366, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:06.873459 9232 init.cc:377] Logged in from keytab as oryx/127.1.210.67@KRBTEST.COM (short username oryx)
I20260806 02:46:06.875275 9232 webserver.cc:460] Webserver started at http://127.1.210.67:37767/ using document root <none> and password file <none>
I20260806 02:46:06.875998 9232 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:06.876120 9232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:06.881822 9232 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:46:06.885108 9247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:06.886790 9232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:46:06.886945 9232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-2/wal
uuid: "04982070d1fd4a2db220580b0b57043c"
format_stamp: "Formatted at 2026-08-06 02:46:02 on dist-test-slave-8tdl"
server_key: "942fd16a549e595f4d3c5a226c271032"
server_key_iv: "5b153645d14e074cf04e06ef1385533c"
server_key_version: "encryptionkey@0"
I20260806 02:46:06.888088 9232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:06.909641 9232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:06.914222 9232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:06.914533 9232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:06.915619 9232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:46:06.917084 9232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:06.917191 9232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:06.917322 9232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:06.917418 9232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:06.963068 9232 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.67:33381
I20260806 02:46:06.963104 9360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.67:33381 every 8 connection(s)
I20260806 02:46:06.965301 9232 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
I20260806 02:46:06.971299 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 9232
Aug 06 02:46:06 dist-test-slave-8tdl krb5kdc[8042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984366, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM
Aug 06 02:46:07 dist-test-slave-8tdl krb5kdc[8042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984366, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.125@KRBTEST.COM
Aug 06 02:46:07 dist-test-slave-8tdl krb5kdc[8042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984366, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.124@KRBTEST.COM
I20260806 02:46:07.025362 9367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:06.970383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:59995 (local address 127.1.210.125:36719)
0806 02:46:06.971077 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:06.971087 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:06.980640 (+ 9553us) server_negotiation.cc:407] Connection header received
0806 02:46:06.980937 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:06.980946 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:06.981046 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:06.981157 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:06.985073 (+ 3916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:06.986252 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.004240 (+ 17988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.005102 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.005272 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:07.011883 (+ 6611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:07.011891 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:07.011898 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:07.011944 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:07.016193 (+ 4249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:07.019336 (+ 3143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:07.019343 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:07.019346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:07.019561 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:07.022616 (+ 3055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:07.022623 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:07.022627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:07.023102 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:07.023293 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:07.024640 (+ 1347us) server_negotiation.cc:299] Negotiation successful
0806 02:46:07.024915 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":293,"threads_started":1}
I20260806 02:46:07.026641 9366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:06.969599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:36719 (local address 127.1.210.67:59995)
0806 02:46:06.980524 (+ 10925us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:06.980548 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:06.980753 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:06.981727 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:06.981740 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:06.982898 (+ 1158us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:06.983947 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:06.983989 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:06.986601 (+ 2612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:06.986609 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:06.991841 (+ 5232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:06.991863 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.005638 (+ 13775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.005647 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:07.005916 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:07.006585 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:07.006616 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:07.011591 (+ 4975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:07.016987 (+ 5396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:07.016996 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:07.017002 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:07.017777 (+ 775us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:07.021060 (+ 3283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:07.021067 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:07.021073 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:07.021260 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:07.023302 (+ 2042us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:07.023314 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:07.023962 (+ 648us) client_negotiation.cc:769] Sending connection context
0806 02:46:07.024334 (+ 372us) client_negotiation.cc:240] Negotiation successful
0806 02:46:07.024984 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"thread_start_us":944,"threads_started":1}
I20260806 02:46:07.027626 9369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:06.971350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.1.210.67:58109)
0806 02:46:06.975932 (+ 4582us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:06.976029 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:06.979508 (+ 3479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:06.987148 (+ 7640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:06.987155 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:06.987750 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:06.988115 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:06.988133 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:06.992961 (+ 4828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:06.992970 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:06.994758 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:06.994778 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:06.996175 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:06.996188 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:06.996601 (+ 413us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:06.997617 (+ 1016us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:06.997670 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:07.002288 (+ 4618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:07.008191 (+ 5903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:07.008206 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:07.008234 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:07.009188 (+ 954us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:07.020448 (+ 11260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:07.020456 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:07.020463 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:07.020880 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:07.026469 (+ 5589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:07.026473 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:07.026657 (+ 184us) client_negotiation.cc:769] Sending connection context
0806 02:46:07.026852 (+ 195us) client_negotiation.cc:240] Negotiation successful
0806 02:46:07.027177 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":946,"threads_started":1}
I20260806 02:46:07.028399 9363 heartbeater.cc:347] Connected to a master server at 127.1.210.125:36719
I20260806 02:46:07.028753 9363 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:07.028973 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:06.970380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:58109 (local address 127.1.210.126:41363)
0806 02:46:06.984668 (+ 14288us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:06.984675 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:06.984698 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:46:06.984765 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:06.984770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:06.984859 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:06.988340 (+ 3481us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:06.988558 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:06.992762 (+ 4204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:06.995048 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:06.995852 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.002633 (+ 6781us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:07.002966 (+ 333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:07.002976 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:07.002984 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:07.003041 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:07.008021 (+ 4980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:07.019728 (+ 11707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:07.019740 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:07.019744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:07.019974 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:07.025888 (+ 5914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:07.025893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:07.025896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:07.026327 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:07.026489 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:07.028215 (+ 1726us) server_negotiation.cc:299] Negotiation successful
0806 02:46:07.028610 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14064,"thread_start_us":13957,"threads_started":1}
I20260806 02:46:07.029783 9363 heartbeater.cc:511] Master 127.1.210.125:36719 requested a full tablet report, sending...
I20260806 02:46:07.030861 9362 heartbeater.cc:347] Connected to a master server at 127.1.210.126:41363
I20260806 02:46:07.030999 9362 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:07.031414 9362 heartbeater.cc:511] Master 127.1.210.126:41363 requested a full tablet report, sending...
I20260806 02:46:07.031956 8702 ts_manager.cc:194] Registered new tserver with Master: 04982070d1fd4a2db220580b0b57043c (127.1.210.67:33381)
I20260806 02:46:07.032641 8088 ts_manager.cc:194] Re-registered known tserver with Master: 04982070d1fd4a2db220580b0b57043c (127.1.210.67:33381)
I20260806 02:46:07.033677 8088 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.210.67@KRBTEST.COM'} at 127.1.210.67:58109
I20260806 02:46:07.034303 9370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:06.972084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:44901 (local address 127.1.210.67:39617)
0806 02:46:06.980111 (+ 8027us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:06.980145 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:06.980392 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:06.997599 (+ 17207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:06.997605 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:06.998405 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:06.998795 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:06.998809 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.000565 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.000571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:07.002724 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:07.002743 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.009027 (+ 6284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.009040 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:07.009302 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:07.010001 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:07.010031 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:07.020988 (+ 10957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:07.025588 (+ 4600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:07.025596 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:07.025600 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:07.026165 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:07.028218 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:07.028223 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:07.028227 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:07.031964 (+ 3737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:07.032875 (+ 911us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:07.032881 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:07.033097 (+ 216us) client_negotiation.cc:769] Sending connection context
0806 02:46:07.033274 (+ 177us) client_negotiation.cc:240] Negotiation successful
0806 02:46:07.033744 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3576,"thread_start_us":3421,"threads_started":1}
I20260806 02:46:07.035619 9365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:06.969113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:39617 (local address 127.1.210.124:44901)
0806 02:46:06.969641 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:06.969648 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:06.997227 (+ 27579us) server_negotiation.cc:407] Connection header received
0806 02:46:06.997301 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:06.997308 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:06.997402 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:06.998944 (+ 1542us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:06.999167 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.000395 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.008022 (+ 7627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.008600 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.008772 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:07.021385 (+ 12613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:07.021395 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:07.021400 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:07.021446 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:07.025416 (+ 3970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:07.026339 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:07.026345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:07.026348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:07.028085 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:07.032188 (+ 4103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:07.032196 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:07.032199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:07.032682 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:07.032884 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:07.034770 (+ 1886us) server_negotiation.cc:299] Negotiation successful
0806 02:46:07.035140 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":266,"threads_started":1}
I20260806 02:46:07.036491 9361 heartbeater.cc:347] Connected to a master server at 127.1.210.124:44901
I20260806 02:46:07.036623 9361 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:07.036962 9361 heartbeater.cc:511] Master 127.1.210.124:44901 requested a full tablet report, sending...
I20260806 02:46:07.038312 8891 ts_manager.cc:194] Registered new tserver with Master: 04982070d1fd4a2db220580b0b57043c (127.1.210.67:33381)
I20260806 02:46:07.045782 9377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:06.976691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41424 (local address 127.1.210.126:41363)
0806 02:46:07.013358 (+ 36667us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:07.013367 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:07.013390 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:46:07.013457 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:07.013463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:07.013547 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:07.013672 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:07.019566 (+ 5894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.021095 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.025327 (+ 4232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.026658 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.026839 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:07.034124 (+ 7285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:07.034131 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:07.034136 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:07.034180 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:07.040880 (+ 6700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:07.042198 (+ 1318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:07.042209 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:07.042215 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:07.042534 (+ 319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:07.043146 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:07.043152 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:07.043156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:07.043688 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:07.044069 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:07.044563 (+ 494us) server_negotiation.cc:299] Negotiation successful
0806 02:46:07.045161 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36516,"thread_start_us":39383,"threads_started":1}
I20260806 02:46:07.049707 9379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:06.996948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45070 (local address 127.1.210.125:36719)
0806 02:46:07.014622 (+ 17674us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:07.014629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:07.014655 (+ 26us) server_negotiation.cc:407] Connection header received
0806 02:46:07.014742 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:07.014748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:07.014831 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:07.014967 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:07.025586 (+ 10619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.029479 (+ 3893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.036270 (+ 6791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.037005 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.037121 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:07.041373 (+ 4252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:07.041386 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:07.041416 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:07.041474 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:07.046357 (+ 4883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:07.047366 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:07.047371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:07.047374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:07.047592 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:07.048033 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:07.048039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:07.048043 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:07.048329 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:07.048605 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:07.048977 (+ 372us) server_negotiation.cc:299] Negotiation successful
0806 02:46:07.049244 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17510,"thread_start_us":17356,"threads_started":1}
I20260806 02:46:07.050884 9378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:06.977022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50012 (local address 127.1.210.124:44901)
0806 02:46:07.012006 (+ 34984us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:07.012014 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:07.012036 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:46:07.012094 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:07.012101 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:07.012204 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:07.013971 (+ 1767us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:07.017869 (+ 3898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.019322 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.024223 (+ 4901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.024940 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.025102 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:07.039005 (+ 13903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:07.039012 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:07.039024 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:07.039076 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:07.044426 (+ 5350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:07.045572 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:07.045579 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:07.045583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:07.045797 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:07.046384 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:07.046394 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:07.046410 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:07.047010 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:07.047394 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:07.049640 (+ 2246us) server_negotiation.cc:299] Negotiation successful
0806 02:46:07.050033 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34836,"thread_start_us":35605,"threads_started":1}
I20260806 02:46:07.054659 8088 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41424:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:46:07.066741 8088 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:46:07.106086 9393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:07.090741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:33381 (local address 127.0.0.1:36808)
0806 02:46:07.094317 (+ 3576us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:07.094345 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:07.094570 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:07.096622 (+ 2052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:07.096627 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:07.096651 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:46:07.097396 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:07.097408 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.099736 (+ 2328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.099741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:07.102110 (+ 2369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:07.102129 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.104890 (+ 2761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.104897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:07.105189 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:07.105216 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 02:46:07.105327 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 02:46:07.105472 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3375,"thread_start_us":8272,"threads_started":1}
I20260806 02:46:07.106525 9392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:07.091220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36808 (local address 127.1.210.67:33381)
0806 02:46:07.093233 (+ 2013us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:07.093243 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:07.095635 (+ 2392us) server_negotiation.cc:407] Connection header received
0806 02:46:07.096056 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:07.096071 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:07.096419 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:07.096663 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:46:07.097620 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.099598 (+ 1978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.102287 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.104572 (+ 2285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.104741 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:07.105090 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:07.105386 (+ 296us) server_negotiation.cc:299] Negotiation successful
0806 02:46:07.105914 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1776,"thread_start_us":1888,"threads_started":1}
I20260806 02:46:07.111745 9296 tablet_service.cc:1467] Processing CreateTablet for tablet e5a4b014e24344e2b993d937ce6f67db (DEFAULT_TABLE table=test-table [id=89e12f782d3a47058fab424cfd487efb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:46:07.113759 9397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:07.094162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51478 (local address 127.1.210.66:34719)
0806 02:46:07.100519 (+ 6357us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:07.100534 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:07.100576 (+ 42us) server_negotiation.cc:407] Connection header received
0806 02:46:07.100714 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:07.100727 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:07.101414 (+ 687us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:07.101742 (+ 328us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:46:07.103374 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.107324 (+ 3950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.110471 (+ 3147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.111684 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.111869 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:07.112189 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:07.113165 (+ 976us) server_negotiation.cc:299] Negotiation successful
0806 02:46:07.113425 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5603,"thread_start_us":5489,"threads_started":1}
I20260806 02:46:07.114145 9396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:07.091068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:42461 (local address 127.0.0.1:52202)
0806 02:46:07.095698 (+ 4630us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:07.095747 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:07.095889 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:07.103813 (+ 7924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:07.103821 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:07.103849 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:46:07.104473 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:07.104483 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.106634 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.106639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:07.109705 (+ 3066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:07.109721 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.112766 (+ 3045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.112780 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:07.113129 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:07.113167 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 02:46:07.113262 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 02:46:07.113438 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4478,"mutex_wait_us":4,"thread_start_us":3550,"threads_started":1}
I20260806 02:46:07.114257 9296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5a4b014e24344e2b993d937ce6f67db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:07.115338 9395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:07.091772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52202 (local address 127.1.210.65:42461)
0806 02:46:07.102910 (+ 11138us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:07.102916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:07.102947 (+ 31us) server_negotiation.cc:407] Connection header received
0806 02:46:07.103023 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:07.103028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:07.103357 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:07.103578 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:46:07.104699 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.106471 (+ 1772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.109937 (+ 3466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.111863 (+ 1926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.112057 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:07.112374 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:07.114825 (+ 2451us) server_negotiation.cc:299] Negotiation successful
0806 02:46:07.115032 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10967,"thread_start_us":10823,"threads_started":1}
I20260806 02:46:07.115336 9391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:07.093382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:34719 (local address 127.0.0.1:51478)
0806 02:46:07.099393 (+ 6011us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:07.099412 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:07.099596 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:07.102336 (+ 2740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:07.102343 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:07.102377 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:46:07.102984 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:07.102999 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.107482 (+ 4483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.107499 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:07.110254 (+ 2755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:07.110266 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.112450 (+ 2184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.112458 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:07.112888 (+ 430us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:07.112931 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 02:46:07.113025 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 02:46:07.114492 (+ 1467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5857,"mutex_wait_us":1365,"thread_start_us":1964,"threads_started":1}
I20260806 02:46:07.119127 9004 tablet_service.cc:1467] Processing CreateTablet for tablet e5a4b014e24344e2b993d937ce6f67db (DEFAULT_TABLE table=test-table [id=89e12f782d3a47058fab424cfd487efb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:46:07.119495 9154 tablet_service.cc:1467] Processing CreateTablet for tablet e5a4b014e24344e2b993d937ce6f67db (DEFAULT_TABLE table=test-table [id=89e12f782d3a47058fab424cfd487efb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:46:07.120657 9004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5a4b014e24344e2b993d937ce6f67db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:07.121127 9154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5a4b014e24344e2b993d937ce6f67db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:07.137678 9398 tablet_bootstrap.cc:492] T e5a4b014e24344e2b993d937ce6f67db P 04982070d1fd4a2db220580b0b57043c: Bootstrap starting.
I20260806 02:46:07.139950 9399 tablet_bootstrap.cc:492] T e5a4b014e24344e2b993d937ce6f67db P cea96d710f5c430db2f412a731bd1bee: Bootstrap starting.
I20260806 02:46:07.142982 9400 tablet_bootstrap.cc:492] T e5a4b014e24344e2b993d937ce6f67db P f4874064dc7841d2b5578c350facdba9: Bootstrap starting.
I20260806 02:46:07.143654 9398 tablet_bootstrap.cc:654] T e5a4b014e24344e2b993d937ce6f67db P 04982070d1fd4a2db220580b0b57043c: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:07.145110 9399 tablet_bootstrap.cc:654] T e5a4b014e24344e2b993d937ce6f67db P cea96d710f5c430db2f412a731bd1bee: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:07.146857 9399 log.cc:824] T e5a4b014e24344e2b993d937ce6f67db P cea96d710f5c430db2f412a731bd1bee: Log is configured to *not* fsync() on all Append() calls
I20260806 02:46:07.146926 9398 log.cc:824] T e5a4b014e24344e2b993d937ce6f67db P 04982070d1fd4a2db220580b0b57043c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:46:07.150741 9400 tablet_bootstrap.cc:654] T e5a4b014e24344e2b993d937ce6f67db P f4874064dc7841d2b5578c350facdba9: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:07.152976 9400 log.cc:824] T e5a4b014e24344e2b993d937ce6f67db P f4874064dc7841d2b5578c350facdba9: Log is configured to *not* fsync() on all Append() calls
I20260806 02:46:07.156553 9398 tablet_bootstrap.cc:492] T e5a4b014e24344e2b993d937ce6f67db P 04982070d1fd4a2db220580b0b57043c: No bootstrap required, opened a new log
I20260806 02:46:07.157033 9398 ts_tablet_manager.cc:1397] T e5a4b014e24344e2b993d937ce6f67db P 04982070d1fd4a2db220580b0b57043c: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.007s
I20260806 02:46:07.158170 9400 tablet_bootstrap.cc:492] T e5a4b014e24344e2b993d937ce6f67db P f4874064dc7841d2b5578c350facdba9: No bootstrap required, opened a new log
I20260806 02:46:07.158547 9400 ts_tablet_manager.cc:1397] T e5a4b014e24344e2b993d937ce6f67db P f4874064dc7841d2b5578c350facdba9: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.009s
I20260806 02:46:07.158586 9399 tablet_bootstrap.cc:492] T e5a4b014e24344e2b993d937ce6f67db P cea96d710f5c430db2f412a731bd1bee: No bootstrap required, opened a new log
I20260806 02:46:07.163340 9399 ts_tablet_manager.cc:1397] T e5a4b014e24344e2b993d937ce6f67db P cea96d710f5c430db2f412a731bd1bee: Time spent bootstrapping tablet: real 0.024s user 0.012s sys 0.005s
I20260806 02:46:07.170209 9400 raft_consensus.cc:348] T e5a4b014e24344e2b993d937ce6f67db P f4874064dc7841d2b5578c350facdba9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04982070d1fd4a2db220580b0b57043c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 33381 } } peers { permanent_uuid: "cea96d710f5c430db2f412a731bd1bee" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 42461 } } peers { permanent_uuid: "f4874064dc7841d2b5578c350facdba9" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 34719 } }
I20260806 02:46:07.170794 9400 raft_consensus.cc:374] T e5a4b014e24344e2b993d937ce6f67db P f4874064dc7841d2b5578c350facdba9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:07.171017 9400 raft_consensus.cc:729] T e5a4b014e24344e2b993d937ce6f67db P f4874064dc7841d2b5578c350facdba9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4874064dc7841d2b5578c350facdba9, State: Initialized, Role: FOLLOWER
I20260806 02:46:07.171942 9400 consensus_queue.cc:260] T e5a4b014e24344e2b993d937ce6f67db P f4874064dc7841d2b5578c350facdba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04982070d1fd4a2db220580b0b57043c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 33381 } } peers { permanent_uuid: "cea96d710f5c430db2f412a731bd1bee" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 42461 } } peers { permanent_uuid: "f4874064dc7841d2b5578c350facdba9" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 34719 } }
I20260806 02:46:07.173408 9398 raft_consensus.cc:348] T e5a4b014e24344e2b993d937ce6f67db P 04982070d1fd4a2db220580b0b57043c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04982070d1fd4a2db220580b0b57043c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 33381 } } peers { permanent_uuid: "cea96d710f5c430db2f412a731bd1bee" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 42461 } } peers { permanent_uuid: "f4874064dc7841d2b5578c350facdba9" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 34719 } }
I20260806 02:46:07.174330 9398 raft_consensus.cc:374] T e5a4b014e24344e2b993d937ce6f67db P 04982070d1fd4a2db220580b0b57043c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:07.174571 9398 raft_consensus.cc:729] T e5a4b014e24344e2b993d937ce6f67db P 04982070d1fd4a2db220580b0b57043c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04982070d1fd4a2db220580b0b57043c, State: Initialized, Role: FOLLOWER
I20260806 02:46:07.174855 9400 ts_tablet_manager.cc:1428] T e5a4b014e24344e2b993d937ce6f67db P f4874064dc7841d2b5578c350facdba9: Time spent starting tablet: real 0.016s user 0.009s sys 0.009s
I20260806 02:46:07.175438 9220 heartbeater.cc:503] Master 127.1.210.126:41363 was elected leader, sending a full tablet report...
I20260806 02:46:07.176764 9398 consensus_queue.cc:260] T e5a4b014e24344e2b993d937ce6f67db P 04982070d1fd4a2db220580b0b57043c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04982070d1fd4a2db220580b0b57043c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 33381 } } peers { permanent_uuid: "cea96d710f5c430db2f412a731bd1bee" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 42461 } } peers { permanent_uuid: "f4874064dc7841d2b5578c350facdba9" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 34719 } }
I20260806 02:46:07.182134 9399 raft_consensus.cc:348] T e5a4b014e24344e2b993d937ce6f67db P cea96d710f5c430db2f412a731bd1bee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04982070d1fd4a2db220580b0b57043c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 33381 } } peers { permanent_uuid: "cea96d710f5c430db2f412a731bd1bee" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 42461 } } peers { permanent_uuid: "f4874064dc7841d2b5578c350facdba9" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 34719 } }
I20260806 02:46:07.183055 9399 raft_consensus.cc:374] T e5a4b014e24344e2b993d937ce6f67db P cea96d710f5c430db2f412a731bd1bee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:07.183283 9399 raft_consensus.cc:729] T e5a4b014e24344e2b993d937ce6f67db P cea96d710f5c430db2f412a731bd1bee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea96d710f5c430db2f412a731bd1bee, State: Initialized, Role: FOLLOWER
I20260806 02:46:07.184401 9399 consensus_queue.cc:260] T e5a4b014e24344e2b993d937ce6f67db P cea96d710f5c430db2f412a731bd1bee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04982070d1fd4a2db220580b0b57043c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 33381 } } peers { permanent_uuid: "cea96d710f5c430db2f412a731bd1bee" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 42461 } } peers { permanent_uuid: "f4874064dc7841d2b5578c350facdba9" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 34719 } }
I20260806 02:46:07.187249 9398 ts_tablet_manager.cc:1428] T e5a4b014e24344e2b993d937ce6f67db P 04982070d1fd4a2db220580b0b57043c: Time spent starting tablet: real 0.030s user 0.017s sys 0.013s
I20260806 02:46:07.187314 9399 ts_tablet_manager.cc:1428] T e5a4b014e24344e2b993d937ce6f67db P cea96d710f5c430db2f412a731bd1bee: Time spent starting tablet: real 0.024s user 0.017s sys 0.005s
I20260806 02:46:07.187803 9362 heartbeater.cc:503] Master 127.1.210.126:41363 was elected leader, sending a full tablet report...
W20260806 02:46:07.220441 9364 tablet.cc:2369] T e5a4b014e24344e2b993d937ce6f67db P 04982070d1fd4a2db220580b0b57043c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:46:07.300827 9222 tablet.cc:2369] T e5a4b014e24344e2b993d937ce6f67db P f4874064dc7841d2b5578c350facdba9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:46:07.343086 9075 tablet.cc:2369] T e5a4b014e24344e2b993d937ce6f67db P cea96d710f5c430db2f412a731bd1bee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:46:07.409214 9406 raft_consensus.cc:482] T e5a4b014e24344e2b993d937ce6f67db P cea96d710f5c430db2f412a731bd1bee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:46:07.409725 9406 raft_consensus.cc:504] T e5a4b014e24344e2b993d937ce6f67db P cea96d710f5c430db2f412a731bd1bee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04982070d1fd4a2db220580b0b57043c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 33381 } } peers { permanent_uuid: "cea96d710f5c430db2f412a731bd1bee" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 42461 } } peers { permanent_uuid: "f4874064dc7841d2b5578c350facdba9" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 34719 } }
I20260806 02:46:07.412110 9406 leader_election.cc:288] T e5a4b014e24344e2b993d937ce6f67db P cea96d710f5c430db2f412a731bd1bee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04982070d1fd4a2db220580b0b57043c (127.1.210.67:33381), f4874064dc7841d2b5578c350facdba9 (127.1.210.66:34719)
I20260806 02:46:07.424296 9392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:07.412488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:52459 (local address 127.1.210.67:33381)
0806 02:46:07.412744 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:07.412753 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:07.415062 (+ 2309us) server_negotiation.cc:407] Connection header received
0806 02:46:07.415391 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:07.415411 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:07.415570 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:07.415756 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:46:07.416498 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.418083 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.421400 (+ 3317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.423167 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.423371 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:07.423436 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:07.423888 (+ 452us) server_negotiation.cc:299] Negotiation successful
0806 02:46:07.424056 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 02:46:07.424510 9407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:07.412649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:33381 (local address 127.1.210.65:52459)
0806 02:46:07.414840 (+ 2191us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:07.414902 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:07.415146 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:07.415787 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:07.415795 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:07.415829 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:46:07.416319 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:07.416349 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.418225 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.418231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:07.421153 (+ 2922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:07.421169 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.423414 (+ 2245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.423421 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:07.423671 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:07.423710 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 02:46:07.423832 (+ 122us) client_negotiation.cc:240] Negotiation successful
0806 02:46:07.424007 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1962,"thread_start_us":1526,"threads_started":1}
I20260806 02:46:07.426255 9316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a4b014e24344e2b993d937ce6f67db" candidate_uuid: "cea96d710f5c430db2f412a731bd1bee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04982070d1fd4a2db220580b0b57043c" is_pre_election: true
I20260806 02:46:07.427039 9397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:07.412768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:56503 (local address 127.1.210.66:34719)
0806 02:46:07.413097 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:07.413108 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:07.416239 (+ 3131us) server_negotiation.cc:407] Connection header received
0806 02:46:07.416423 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:07.416431 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:07.416524 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:07.416592 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:46:07.417952 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.419585 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.423363 (+ 3778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.425438 (+ 2075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.425585 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:07.425639 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:07.426431 (+ 792us) server_negotiation.cc:299] Negotiation successful
0806 02:46:07.426550 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260806 02:46:07.427165 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:07.414187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:34719 (local address 127.1.210.65:56503)
0806 02:46:07.416121 (+ 1934us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:07.416138 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:07.416280 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:07.416663 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:07.416668 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:07.416705 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:46:07.417655 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:07.417671 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.419807 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.419811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:07.423126 (+ 3315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:07.423144 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.425867 (+ 2723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.425876 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:07.426186 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:07.426246 (+ 60us) client_negotiation.cc:769] Sending connection context
0806 02:46:07.426365 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 02:46:07.426575 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1682,"thread_start_us":1257,"threads_started":1}
I20260806 02:46:07.427310 9316 raft_consensus.cc:2454] T e5a4b014e24344e2b993d937ce6f67db P 04982070d1fd4a2db220580b0b57043c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea96d710f5c430db2f412a731bd1bee in term 0.
I20260806 02:46:07.428321 8958 leader_election.cc:302] T e5a4b014e24344e2b993d937ce6f67db P cea96d710f5c430db2f412a731bd1bee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04982070d1fd4a2db220580b0b57043c, cea96d710f5c430db2f412a731bd1bee; no voters:
I20260806 02:46:07.429055 9406 raft_consensus.cc:2781] T e5a4b014e24344e2b993d937ce6f67db P cea96d710f5c430db2f412a731bd1bee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:46:07.428895 9174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a4b014e24344e2b993d937ce6f67db" candidate_uuid: "cea96d710f5c430db2f412a731bd1bee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4874064dc7841d2b5578c350facdba9" is_pre_election: true
I20260806 02:46:07.429313 9406 raft_consensus.cc:482] T e5a4b014e24344e2b993d937ce6f67db P cea96d710f5c430db2f412a731bd1bee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:46:07.429527 9406 raft_consensus.cc:3037] T e5a4b014e24344e2b993d937ce6f67db P cea96d710f5c430db2f412a731bd1bee [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:07.429749 9174 raft_consensus.cc:2454] T e5a4b014e24344e2b993d937ce6f67db P f4874064dc7841d2b5578c350facdba9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea96d710f5c430db2f412a731bd1bee in term 0.
I20260806 02:46:07.432881 9406 raft_consensus.cc:504] T e5a4b014e24344e2b993d937ce6f67db P cea96d710f5c430db2f412a731bd1bee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04982070d1fd4a2db220580b0b57043c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 33381 } } peers { permanent_uuid: "cea96d710f5c430db2f412a731bd1bee" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 42461 } } peers { permanent_uuid: "f4874064dc7841d2b5578c350facdba9" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 34719 } }
I20260806 02:46:07.433614 9406 leader_election.cc:288] T e5a4b014e24344e2b993d937ce6f67db P cea96d710f5c430db2f412a731bd1bee [CANDIDATE]: Term 1 election: Requested vote from peers 04982070d1fd4a2db220580b0b57043c (127.1.210.67:33381), f4874064dc7841d2b5578c350facdba9 (127.1.210.66:34719)
I20260806 02:46:07.434507 9174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a4b014e24344e2b993d937ce6f67db" candidate_uuid: "cea96d710f5c430db2f412a731bd1bee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4874064dc7841d2b5578c350facdba9"
I20260806 02:46:07.434520 9316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a4b014e24344e2b993d937ce6f67db" candidate_uuid: "cea96d710f5c430db2f412a731bd1bee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04982070d1fd4a2db220580b0b57043c"
I20260806 02:46:07.434849 9316 raft_consensus.cc:3037] T e5a4b014e24344e2b993d937ce6f67db P 04982070d1fd4a2db220580b0b57043c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:07.434841 9174 raft_consensus.cc:3037] T e5a4b014e24344e2b993d937ce6f67db P f4874064dc7841d2b5578c350facdba9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:07.439798 9316 raft_consensus.cc:2454] T e5a4b014e24344e2b993d937ce6f67db P 04982070d1fd4a2db220580b0b57043c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea96d710f5c430db2f412a731bd1bee in term 1.
I20260806 02:46:07.440320 9174 raft_consensus.cc:2454] T e5a4b014e24344e2b993d937ce6f67db P f4874064dc7841d2b5578c350facdba9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea96d710f5c430db2f412a731bd1bee in term 1.
I20260806 02:46:07.440603 8958 leader_election.cc:302] T e5a4b014e24344e2b993d937ce6f67db P cea96d710f5c430db2f412a731bd1bee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04982070d1fd4a2db220580b0b57043c, cea96d710f5c430db2f412a731bd1bee; no voters:
I20260806 02:46:07.441268 9406 raft_consensus.cc:2781] T e5a4b014e24344e2b993d937ce6f67db P cea96d710f5c430db2f412a731bd1bee [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:46:07.442126 9406 raft_consensus.cc:686] T e5a4b014e24344e2b993d937ce6f67db P cea96d710f5c430db2f412a731bd1bee [term 1 LEADER]: Becoming Leader. State: Replica: cea96d710f5c430db2f412a731bd1bee, State: Running, Role: LEADER
I20260806 02:46:07.442996 9406 consensus_queue.cc:237] T e5a4b014e24344e2b993d937ce6f67db P cea96d710f5c430db2f412a731bd1bee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04982070d1fd4a2db220580b0b57043c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 33381 } } peers { permanent_uuid: "cea96d710f5c430db2f412a731bd1bee" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 42461 } } peers { permanent_uuid: "f4874064dc7841d2b5578c350facdba9" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 34719 } }
I20260806 02:46:07.449832 8087 catalog_manager.cc:5495] T e5a4b014e24344e2b993d937ce6f67db P cea96d710f5c430db2f412a731bd1bee reported cstate change: term changed from 0 to 1, leader changed from <none> to cea96d710f5c430db2f412a731bd1bee (127.1.210.65). New cstate: current_term: 1 leader_uuid: "cea96d710f5c430db2f412a731bd1bee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04982070d1fd4a2db220580b0b57043c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 33381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cea96d710f5c430db2f412a731bd1bee" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 42461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4874064dc7841d2b5578c350facdba9" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 34719 } health_report { overall_health: UNKNOWN } } }
I20260806 02:46:07.489044 9395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:07.481694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52210 (local address 127.1.210.65:42461)
0806 02:46:07.482012 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:07.482017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:07.482034 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:46:07.482134 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:07.482137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:07.482254 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:07.482434 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:46:07.483017 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.484128 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.486051 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.486853 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.486968 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:07.487655 (+ 687us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:46:07.488345 (+ 690us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:46:07.488485 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:07.488620 (+ 135us) server_negotiation.cc:299] Negotiation successful
0806 02:46:07.488703 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 02:46:07.500705 9316 raft_consensus.cc:1261] T e5a4b014e24344e2b993d937ce6f67db P 04982070d1fd4a2db220580b0b57043c [term 1 FOLLOWER]: Refusing update from remote peer cea96d710f5c430db2f412a731bd1bee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:46:07.500862 9174 raft_consensus.cc:1261] T e5a4b014e24344e2b993d937ce6f67db P f4874064dc7841d2b5578c350facdba9 [term 1 FOLLOWER]: Refusing update from remote peer cea96d710f5c430db2f412a731bd1bee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:46:07.502074 9409 consensus_queue.cc:1035] T e5a4b014e24344e2b993d937ce6f67db P cea96d710f5c430db2f412a731bd1bee [LEADER]: Connected to new peer: Peer: permanent_uuid: "04982070d1fd4a2db220580b0b57043c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 33381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:46:07.502791 9406 consensus_queue.cc:1035] T e5a4b014e24344e2b993d937ce6f67db P cea96d710f5c430db2f412a731bd1bee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4874064dc7841d2b5578c350facdba9" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 34719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:46:07.553851 8088 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:46:07.554368 8088 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:46:07.563069 8088 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 869bf3f6f6ad40e5ba34f73d3004a761: Sending DeleteTablet for 3 replicas of tablet e5a4b014e24344e2b993d937ce6f67db
I20260806 02:46:07.564448 9296 tablet_service.cc:1514] Processing DeleteTablet for tablet e5a4b014e24344e2b993d937ce6f67db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:46:07 UTC) from {username='oryx', principal='oryx/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:36808
I20260806 02:46:07.564532 9004 tablet_service.cc:1514] Processing DeleteTablet for tablet e5a4b014e24344e2b993d937ce6f67db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:46:07 UTC) from {username='oryx', principal='oryx/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:52202
I20260806 02:46:07.564821 9154 tablet_service.cc:1514] Processing DeleteTablet for tablet e5a4b014e24344e2b993d937ce6f67db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:46:07 UTC) from {username='oryx', principal='oryx/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:51478
I20260806 02:46:07.575577 9424 tablet_replica.cc:330] T e5a4b014e24344e2b993d937ce6f67db P f4874064dc7841d2b5578c350facdba9: stopping tablet replica
I20260806 02:46:07.576443 9424 raft_consensus.cc:2229] T e5a4b014e24344e2b993d937ce6f67db P f4874064dc7841d2b5578c350facdba9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:46:07.577689 9424 raft_consensus.cc:2258] T e5a4b014e24344e2b993d937ce6f67db P f4874064dc7841d2b5578c350facdba9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:46:07.581389 9423 tablet_replica.cc:330] T e5a4b014e24344e2b993d937ce6f67db P 04982070d1fd4a2db220580b0b57043c: stopping tablet replica
I20260806 02:46:07.582036 9423 raft_consensus.cc:2229] T e5a4b014e24344e2b993d937ce6f67db P 04982070d1fd4a2db220580b0b57043c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:46:07.583412 9423 raft_consensus.cc:2258] T e5a4b014e24344e2b993d937ce6f67db P 04982070d1fd4a2db220580b0b57043c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:46:07.585928 9424 ts_tablet_manager.cc:1905] T e5a4b014e24344e2b993d937ce6f67db P f4874064dc7841d2b5578c350facdba9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:46:07.593701 9423 ts_tablet_manager.cc:1905] T e5a4b014e24344e2b993d937ce6f67db P 04982070d1fd4a2db220580b0b57043c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:46:07.595110 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 8055
I20260806 02:46:07.596727 8119 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:46:07.609652 9424 ts_tablet_manager.cc:1918] T e5a4b014e24344e2b993d937ce6f67db P f4874064dc7841d2b5578c350facdba9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:46:07.609771 9422 tablet_replica.cc:330] T e5a4b014e24344e2b993d937ce6f67db P cea96d710f5c430db2f412a731bd1bee: stopping tablet replica
I20260806 02:46:07.609980 9424 log.cc:1196] T e5a4b014e24344e2b993d937ce6f67db P f4874064dc7841d2b5578c350facdba9: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-1/wal/wals/e5a4b014e24344e2b993d937ce6f67db
I20260806 02:46:07.610513 9422 raft_consensus.cc:2229] T e5a4b014e24344e2b993d937ce6f67db P cea96d710f5c430db2f412a731bd1bee [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:46:07.610888 9424 ts_tablet_manager.cc:1939] T e5a4b014e24344e2b993d937ce6f67db P f4874064dc7841d2b5578c350facdba9: Deleting consensus metadata
I20260806 02:46:07.611812 9422 raft_consensus.cc:2258] T e5a4b014e24344e2b993d937ce6f67db P cea96d710f5c430db2f412a731bd1bee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:46:07.615232 9423 ts_tablet_manager.cc:1918] T e5a4b014e24344e2b993d937ce6f67db P 04982070d1fd4a2db220580b0b57043c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:46:07.615480 9423 log.cc:1196] T e5a4b014e24344e2b993d937ce6f67db P 04982070d1fd4a2db220580b0b57043c: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-2/wal/wals/e5a4b014e24344e2b993d937ce6f67db
I20260806 02:46:07.616173 9423 ts_tablet_manager.cc:1939] T e5a4b014e24344e2b993d937ce6f67db P 04982070d1fd4a2db220580b0b57043c: Deleting consensus metadata
I20260806 02:46:07.625221 9422 ts_tablet_manager.cc:1905] T e5a4b014e24344e2b993d937ce6f67db P cea96d710f5c430db2f412a731bd1bee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:46:07.640225 9422 ts_tablet_manager.cc:1918] T e5a4b014e24344e2b993d937ce6f67db P cea96d710f5c430db2f412a731bd1bee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:46:07.640549 9422 log.cc:1196] T e5a4b014e24344e2b993d937ce6f67db P cea96d710f5c430db2f412a731bd1bee: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-0/wal/wals/e5a4b014e24344e2b993d937ce6f67db
I20260806 02:46:07.641613 9422 ts_tablet_manager.cc:1939] T e5a4b014e24344e2b993d937ce6f67db P cea96d710f5c430db2f412a731bd1bee: Deleting consensus metadata
I20260806 02:46:07.648294 9426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:07.574476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45076 (local address 127.1.210.125:36719)
0806 02:46:07.575409 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:07.575420 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:07.586990 (+ 11570us) server_negotiation.cc:407] Connection header received
0806 02:46:07.600871 (+ 13881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:07.600882 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:07.601005 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:07.601146 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:07.608729 (+ 7583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.610444 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.625794 (+ 15350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.626737 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.626898 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:07.637804 (+ 10906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:07.637822 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:07.637828 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:07.637882 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:07.643221 (+ 5339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:07.644570 (+ 1349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:07.644576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:07.644580 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:07.644863 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:07.645531 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:07.645536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:07.645540 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:07.645927 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:07.646477 (+ 550us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:07.647276 (+ 799us) server_negotiation.cc:299] Negotiation successful
0806 02:46:07.647604 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":578,"threads_started":1}
I20260806 02:46:07.671181 9427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:07.587731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50014 (local address 127.1.210.124:44901)
0806 02:46:07.601038 (+ 13307us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:07.601045 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:07.648377 (+ 47332us) server_negotiation.cc:407] Connection header received
0806 02:46:07.648680 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:07.648687 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:07.648780 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:07.648867 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:07.651270 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.652069 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.655279 (+ 3210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.656552 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.656711 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:07.663086 (+ 6375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:07.663092 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:07.663096 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:07.663172 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:07.667510 (+ 4338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:07.668490 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:07.668504 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:07.668507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:07.668711 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:07.669250 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:07.669254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:07.669257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:07.669616 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:07.669997 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:07.670318 (+ 321us) server_negotiation.cc:299] Negotiation successful
0806 02:46:07.670611 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13151,"thread_start_us":12607,"threads_started":1}
I20260806 02:46:07.830756 8073 catalog_manager.cc:4841] TS f4874064dc7841d2b5578c350facdba9 (127.1.210.66:34719): tablet e5a4b014e24344e2b993d937ce6f67db (table test-table [id=89e12f782d3a47058fab424cfd487efb]) successfully deleted
I20260806 02:46:07.831094 8072 catalog_manager.cc:4841] TS 04982070d1fd4a2db220580b0b57043c (127.1.210.67:33381): tablet e5a4b014e24344e2b993d937ce6f67db (table test-table [id=89e12f782d3a47058fab424cfd487efb]) successfully deleted
I20260806 02:46:07.831485 8071 catalog_manager.cc:4841] TS cea96d710f5c430db2f412a731bd1bee (127.1.210.65:42461): tablet e5a4b014e24344e2b993d937ce6f67db (table test-table [id=89e12f782d3a47058fab424cfd487efb]) successfully deleted
I20260806 02:46:07.831701 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 8055
I20260806 02:46:07.896473 9426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:07.871601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45092 (local address 127.1.210.125:36719)
0806 02:46:07.871922 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:07.871930 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:07.875568 (+ 3638us) server_negotiation.cc:407] Connection header received
0806 02:46:07.875647 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:07.875652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:07.875772 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:07.875878 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:07.879887 (+ 4009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.881175 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.883296 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.883756 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.883891 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:07.888761 (+ 4870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:07.888769 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:07.888774 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:07.888838 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:07.892660 (+ 3822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:07.893708 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:07.893718 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:07.893722 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:07.893918 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:07.894422 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:07.894428 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:07.894432 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:07.894862 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:07.895186 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:07.895585 (+ 399us) server_negotiation.cc:299] Negotiation successful
0806 02:46:07.896056 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260806 02:46:07.897634 9427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:07.871592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50024 (local address 127.1.210.124:44901)
0806 02:46:07.871920 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:07.871927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:07.874535 (+ 2608us) server_negotiation.cc:407] Connection header received
0806 02:46:07.875116 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:07.875122 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:07.875219 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:07.875301 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:07.880533 (+ 5232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.881289 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.883831 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.884850 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.885018 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:07.890799 (+ 5781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:07.890807 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:07.890810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:07.890851 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:07.894434 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:07.895341 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:07.895347 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:07.895351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:07.895529 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:07.895963 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:07.895968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:07.895972 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:07.896253 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:07.896439 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:07.897043 (+ 604us) server_negotiation.cc:299] Negotiation successful
0806 02:46:07.897285 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 02:46:08.544631 9436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:08.543279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.1.210.67:37145)
0806 02:46:08.544234 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:08.544393 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:41363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":842,"thread_start_us":630,"threads_started":1}
I20260806 02:46:08.544998 9437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:08.544032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.1.210.65:34137)
0806 02:46:08.544546 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:08.544723 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:41363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"spinlock_wait_cycles":6912,"thread_start_us":326,"threads_started":1}
W20260806 02:46:08.545195 9362 heartbeater.cc:644] Failed to heartbeat to 127.1.210.126:41363 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.210.126:41363: connect: Connection refused (error 111)
W20260806 02:46:08.545400 9071 heartbeater.cc:644] Failed to heartbeat to 127.1.210.126:41363 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.210.126:41363: connect: Connection refused (error 111)
I20260806 02:46:08.551393 9438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:08.546411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.1.210.66:51449)
0806 02:46:08.551101 (+ 4690us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:08.551233 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:41363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4578,"thread_start_us":4459,"threads_started":1}
W20260806 02:46:08.551872 9220 heartbeater.cc:644] Failed to heartbeat to 127.1.210.126:41363 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.210.126:41363: connect: Connection refused (error 111)
I20260806 02:46:09.066038 9439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869bf3f6f6ad40e5ba34f73d3004a761)
I20260806 02:46:09.066372 9439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false } } peers { permanent_uuid: "9cd0128d2476470cb51e77201cfb727a" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44901 } attrs { promote: false } }
I20260806 02:46:09.067917 9439 leader_election.cc:288] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869bf3f6f6ad40e5ba34f73d3004a761 (127.1.210.126:41363), 88068f45c0904cde8fc22a8e54bc94c0 (127.1.210.125:36719)
I20260806 02:46:09.069499 9441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.068130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.0.0.1:41818)
0806 02:46:09.068841 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:09.069126 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:41363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":538,"threads_started":1}
W20260806 02:46:09.069875 8875 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.210.126:41363: connect: Connection refused (error 111)
I20260806 02:46:09.071557 9444 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869bf3f6f6ad40e5ba34f73d3004a761)
I20260806 02:46:09.071671 9441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.071268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.0.0.1:41834)
0806 02:46:09.071434 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:09.071527 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:41363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260806 02:46:09.071918 8875 leader_election.cc:334] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 869bf3f6f6ad40e5ba34f73d3004a761 (127.1.210.126:41363): Network error: Client connection negotiation failed: client connection to 127.1.210.126:41363: connect: Connection refused (error 111)
I20260806 02:46:09.071923 9444 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false } } peers { permanent_uuid: "9cd0128d2476470cb51e77201cfb727a" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44901 } attrs { promote: false } }
I20260806 02:46:09.073807 9444 leader_election.cc:288] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869bf3f6f6ad40e5ba34f73d3004a761 (127.1.210.126:41363), 9cd0128d2476470cb51e77201cfb727a (127.1.210.124:44901)
I20260806 02:46:09.077913 9440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.068130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:36719 (local address 127.0.0.1:45098)
0806 02:46:09.069757 (+ 1627us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:09.069773 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:09.069916 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:09.070345 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:09.070354 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:09.070398 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:46:09.070827 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:09.070849 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.072399 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.072404 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:09.073949 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:09.073967 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.076619 (+ 2652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.076628 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:09.076903 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:09.076960 (+ 57us) client_negotiation.cc:769] Sending connection context
0806 02:46:09.077089 (+ 129us) client_negotiation.cc:240] Negotiation successful
0806 02:46:09.077297 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1483,"mutex_wait_us":28,"thread_start_us":262,"threads_started":1}
I20260806 02:46:09.079020 9442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.068266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45098 (local address 127.1.210.125:36719)
0806 02:46:09.069707 (+ 1441us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:09.069716 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:09.069881 (+ 165us) server_negotiation.cc:407] Connection header received
0806 02:46:09.070058 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:09.070063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:09.070148 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:09.070260 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:46:09.071060 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.072185 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.074163 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.076127 (+ 1964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.076298 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:09.076471 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:09.078218 (+ 1747us) server_negotiation.cc:299] Negotiation successful
0806 02:46:09.078437 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1143,"thread_start_us":786,"threads_started":1}
I20260806 02:46:09.080262 8712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9cd0128d2476470cb51e77201cfb727a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "88068f45c0904cde8fc22a8e54bc94c0" is_pre_election: true
I20260806 02:46:09.080744 8712 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cd0128d2476470cb51e77201cfb727a in term 1.
I20260806 02:46:09.080891 9446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.073925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.0.0.1:41848)
0806 02:46:09.080255 (+ 6330us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:09.080584 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:41363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6181,"threads_started":1}
I20260806 02:46:09.081337 8878 leader_election.cc:302] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88068f45c0904cde8fc22a8e54bc94c0, 9cd0128d2476470cb51e77201cfb727a; no voters: 869bf3f6f6ad40e5ba34f73d3004a761
W20260806 02:46:09.081498 8686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.210.126:41363: connect: Connection refused (error 111)
I20260806 02:46:09.081833 9439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 02:46:09.081990 9439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869bf3f6f6ad40e5ba34f73d3004a761)
I20260806 02:46:09.082202 9439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:46:09.083380 9446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.082950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.0.0.1:41856)
0806 02:46:09.083121 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:09.083214 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:41363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260806 02:46:09.084633 8686 leader_election.cc:334] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 869bf3f6f6ad40e5ba34f73d3004a761 (127.1.210.126:41363): Network error: Client connection negotiation failed: client connection to 127.1.210.126:41363: connect: Connection refused (error 111)
I20260806 02:46:09.085568 9439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false } } peers { permanent_uuid: "9cd0128d2476470cb51e77201cfb727a" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44901 } attrs { promote: false } }
I20260806 02:46:09.086539 9439 leader_election.cc:288] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [CANDIDATE]: Term 2 election: Requested vote from peers 869bf3f6f6ad40e5ba34f73d3004a761 (127.1.210.126:41363), 88068f45c0904cde8fc22a8e54bc94c0 (127.1.210.125:36719)
I20260806 02:46:09.087109 9440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.086458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.0.0.1:41860)
0806 02:46:09.086852 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:09.086940 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:41363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282}
I20260806 02:46:09.087821 8712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9cd0128d2476470cb51e77201cfb727a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "88068f45c0904cde8fc22a8e54bc94c0"
I20260806 02:46:09.088033 8712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:46:09.088495 9440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.088116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.0.0.1:41878)
0806 02:46:09.088309 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:09.088379 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:41363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260806 02:46:09.088662 8875 leader_election.cc:334] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 869bf3f6f6ad40e5ba34f73d3004a761 (127.1.210.126:41363): Network error: Client connection negotiation failed: client connection to 127.1.210.126:41363: connect: Connection refused (error 111)
I20260806 02:46:09.090798 8712 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cd0128d2476470cb51e77201cfb727a in term 2.
I20260806 02:46:09.091470 8878 leader_election.cc:302] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88068f45c0904cde8fc22a8e54bc94c0, 9cd0128d2476470cb51e77201cfb727a; no voters: 869bf3f6f6ad40e5ba34f73d3004a761
I20260806 02:46:09.091807 9439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [term 2 FOLLOWER]: Leader election won for term 2
I20260806 02:46:09.092407 9439 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [term 2 LEADER]: Becoming Leader. State: Replica: 9cd0128d2476470cb51e77201cfb727a, State: Running, Role: LEADER
I20260806 02:46:09.092882 9439 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false } } peers { permanent_uuid: "9cd0128d2476470cb51e77201cfb727a" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44901 } attrs { promote: false } }
I20260806 02:46:09.093693 9447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.074558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50036 (local address 127.1.210.124:44901)
0806 02:46:09.079757 (+ 5199us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:09.079767 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:09.080534 (+ 767us) server_negotiation.cc:407] Connection header received
0806 02:46:09.080846 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:09.080853 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:09.081031 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:09.081152 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:46:09.081872 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.083208 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.085740 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.092481 (+ 6741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.092668 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:09.092794 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:09.093121 (+ 327us) server_negotiation.cc:299] Negotiation successful
0806 02:46:09.093270 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5042,"thread_start_us":4859,"threads_started":1}
I20260806 02:46:09.093710 9445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.074021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:44901 (local address 127.0.0.1:50036)
0806 02:46:09.080403 (+ 6382us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:09.080431 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:09.080684 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:09.081222 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:09.081227 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:09.081258 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:46:09.081637 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:09.081650 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.083381 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.083386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:09.085562 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:09.085580 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.092715 (+ 7135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.092722 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:09.092943 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:09.092976 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 02:46:09.093096 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 02:46:09.093272 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6227,"spinlock_wait_cycles":2304,"thread_start_us":6202,"threads_started":1}
I20260806 02:46:09.094585 8901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88068f45c0904cde8fc22a8e54bc94c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9cd0128d2476470cb51e77201cfb727a" is_pre_election: true
I20260806 02:46:09.095587 8687 leader_election.cc:302] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88068f45c0904cde8fc22a8e54bc94c0; no voters: 869bf3f6f6ad40e5ba34f73d3004a761, 9cd0128d2476470cb51e77201cfb727a
I20260806 02:46:09.095386 9449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cd0128d2476470cb51e77201cfb727a. Latest consensus state: current_term: 2 leader_uuid: "9cd0128d2476470cb51e77201cfb727a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false } } peers { permanent_uuid: "9cd0128d2476470cb51e77201cfb727a" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44901 } attrs { promote: false } } }
I20260806 02:46:09.095912 9449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [sys.catalog]: This master's current role is: LEADER
I20260806 02:46:09.095981 9444 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:46:09.096621 9451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:46:09.100776 9451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:46:09.101696 9451 catalog_manager.cc:1259] Loaded cluster ID: fa59af352fb5431abc8784c5d36b6618
I20260806 02:46:09.101769 9451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:46:09.102972 9451 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:46:09.103751 9451 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a: Loaded TSK: 0
I20260806 02:46:09.104703 9451 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:46:09.151859 8890 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:46:09.159889 8890 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:46:09.164565 9440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.164078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.0.0.1:41902)
0806 02:46:09.164274 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:09.164414 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:41363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260806 02:46:09.164865 8712 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [term 2 FOLLOWER]: Refusing update from remote peer 9cd0128d2476470cb51e77201cfb727a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260806 02:46:09.165792 9440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.165326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.0.0.1:41918)
0806 02:46:09.165521 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:09.165624 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:41363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260806 02:46:09.165753 9449 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [LEADER]: Connected to new peer: Peer: permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260806 02:46:09.166080 8875 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a -> Peer 869bf3f6f6ad40e5ba34f73d3004a761 (127.1.210.126:41363): Couldn't send request to peer 869bf3f6f6ad40e5ba34f73d3004a761. Status: Network error: Client connection negotiation failed: client connection to 127.1.210.126:41363: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 02:46:09.168831 9444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cd0128d2476470cb51e77201cfb727a. Latest consensus state: current_term: 2 leader_uuid: "9cd0128d2476470cb51e77201cfb727a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false } } peers { permanent_uuid: "9cd0128d2476470cb51e77201cfb727a" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44901 } attrs { promote: false } } }
I20260806 02:46:09.169173 9444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:46:09.170933 9449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "9cd0128d2476470cb51e77201cfb727a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false } } peers { permanent_uuid: "9cd0128d2476470cb51e77201cfb727a" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44901 } attrs { promote: false } } }
I20260806 02:46:09.171356 9449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a [sys.catalog]: This master's current role is: LEADER
I20260806 02:46:09.171628 9444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "9cd0128d2476470cb51e77201cfb727a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "869bf3f6f6ad40e5ba34f73d3004a761" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41363 } } peers { permanent_uuid: "88068f45c0904cde8fc22a8e54bc94c0" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 36719 } attrs { promote: false } } peers { permanent_uuid: "9cd0128d2476470cb51e77201cfb727a" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44901 } attrs { promote: false } } }
I20260806 02:46:09.171903 9444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88068f45c0904cde8fc22a8e54bc94c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:46:09.172771 9452 mvcc.cc:205] Tried to move back new op lower bound from 7315391976090718208 to 7315391975809093632. Current Snapshot: MvccSnapshot[applied={T|T < 7315391976090718208 or (T in {7315391976090718208})}]
I20260806 02:46:09.198211 9440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.181642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:42461 (local address 127.0.0.1:52220)
0806 02:46:09.181833 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:09.181852 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:09.182067 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:09.183659 (+ 1592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:09.183668 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:09.183708 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:46:09.184020 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:09.184037 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.190192 (+ 6155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.190203 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:09.193367 (+ 3164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:09.193403 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.197302 (+ 3899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.197309 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:09.197531 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:09.197574 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 02:46:09.197703 (+ 129us) client_negotiation.cc:240] Negotiation successful
0806 02:46:09.197860 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 02:46:09.198433 9457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.182017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52220 (local address 127.1.210.65:42461)
0806 02:46:09.183126 (+ 1109us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:09.183134 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:09.183171 (+ 37us) server_negotiation.cc:407] Connection header received
0806 02:46:09.183271 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:09.183280 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:09.183400 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:09.185962 (+ 2562us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:46:09.186292 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.188124 (+ 1832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.194701 (+ 6577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.197043 (+ 2342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.197276 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:09.197511 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:09.197769 (+ 258us) server_negotiation.cc:299] Negotiation successful
0806 02:46:09.198010 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":969,"thread_start_us":1068,"threads_started":1}
I20260806 02:46:09.199637 9004 tablet_service.cc:1467] Processing CreateTablet for tablet 97ac65438b414347b0861dee921602a4 (DEFAULT_TABLE table=test-table [id=f7784dab20a745e0aee2464a06077d55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:46:09.200177 9004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97ac65438b414347b0861dee921602a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:09.205740 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.182931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36814 (local address 127.1.210.67:33381)
0806 02:46:09.193706 (+ 10775us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:09.193715 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:09.194084 (+ 369us) server_negotiation.cc:407] Connection header received
0806 02:46:09.194308 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:09.194313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:09.194393 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:09.194522 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:46:09.195439 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.196187 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.198307 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.203729 (+ 5422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.203950 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:09.204077 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:09.205237 (+ 1160us) server_negotiation.cc:299] Negotiation successful
0806 02:46:09.205428 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10606,"thread_start_us":10479,"threads_started":1}
I20260806 02:46:09.205770 9458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.182852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:33381 (local address 127.0.0.1:36814)
0806 02:46:09.193943 (+ 11091us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:09.193972 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:09.194165 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:09.194759 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:09.194763 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:09.194787 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:46:09.195215 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:09.195231 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.196342 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.196349 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:09.198085 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:09.198098 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.204272 (+ 6174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.204282 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:09.204818 (+ 536us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:09.204854 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 02:46:09.205086 (+ 232us) client_negotiation.cc:240] Negotiation successful
0806 02:46:09.205380 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10961,"thread_start_us":10853,"threads_started":1}
I20260806 02:46:09.206588 9441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.182316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:34719 (local address 127.0.0.1:51488)
0806 02:46:09.182537 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:09.182557 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:09.182700 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:09.199143 (+ 16443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:09.199150 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:09.199203 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:46:09.199508 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:09.199520 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.201074 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.201080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:09.203406 (+ 2326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:09.203428 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.205445 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.205453 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:09.205809 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:09.205847 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 02:46:09.205940 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 02:46:09.206190 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260806 02:46:09.207126 9296 tablet_service.cc:1467] Processing CreateTablet for tablet 97ac65438b414347b0861dee921602a4 (DEFAULT_TABLE table=test-table [id=f7784dab20a745e0aee2464a06077d55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:46:09.207607 9296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97ac65438b414347b0861dee921602a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:09.208760 9461 tablet_bootstrap.cc:492] T 97ac65438b414347b0861dee921602a4 P cea96d710f5c430db2f412a731bd1bee: Bootstrap starting.
I20260806 02:46:09.210963 9461 tablet_bootstrap.cc:654] T 97ac65438b414347b0861dee921602a4 P cea96d710f5c430db2f412a731bd1bee: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:09.211627 9460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.182975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51488 (local address 127.1.210.66:34719)
0806 02:46:09.198487 (+ 15512us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:09.198495 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:09.198520 (+ 25us) server_negotiation.cc:407] Connection header received
0806 02:46:09.198614 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:09.198620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:09.198790 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:09.199095 (+ 305us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:46:09.199698 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.200711 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.203660 (+ 2949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.204919 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.206513 (+ 1594us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:09.206634 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:09.210973 (+ 4339us) server_negotiation.cc:299] Negotiation successful
0806 02:46:09.211269 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15344,"thread_start_us":15931,"threads_started":1}
I20260806 02:46:09.212370 9154 tablet_service.cc:1467] Processing CreateTablet for tablet 97ac65438b414347b0861dee921602a4 (DEFAULT_TABLE table=test-table [id=f7784dab20a745e0aee2464a06077d55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:46:09.213096 9154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97ac65438b414347b0861dee921602a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:09.214928 9463 tablet_bootstrap.cc:492] T 97ac65438b414347b0861dee921602a4 P 04982070d1fd4a2db220580b0b57043c: Bootstrap starting.
I20260806 02:46:09.216800 9463 tablet_bootstrap.cc:654] T 97ac65438b414347b0861dee921602a4 P 04982070d1fd4a2db220580b0b57043c: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:09.216903 9461 tablet_bootstrap.cc:492] T 97ac65438b414347b0861dee921602a4 P cea96d710f5c430db2f412a731bd1bee: No bootstrap required, opened a new log
I20260806 02:46:09.217094 9461 ts_tablet_manager.cc:1397] T 97ac65438b414347b0861dee921602a4 P cea96d710f5c430db2f412a731bd1bee: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260806 02:46:09.218046 9461 raft_consensus.cc:348] T 97ac65438b414347b0861dee921602a4 P cea96d710f5c430db2f412a731bd1bee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea96d710f5c430db2f412a731bd1bee" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 42461 } } peers { permanent_uuid: "f4874064dc7841d2b5578c350facdba9" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 34719 } } peers { permanent_uuid: "04982070d1fd4a2db220580b0b57043c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 33381 } }
I20260806 02:46:09.218324 9461 raft_consensus.cc:374] T 97ac65438b414347b0861dee921602a4 P cea96d710f5c430db2f412a731bd1bee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:09.218439 9461 raft_consensus.cc:729] T 97ac65438b414347b0861dee921602a4 P cea96d710f5c430db2f412a731bd1bee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea96d710f5c430db2f412a731bd1bee, State: Initialized, Role: FOLLOWER
I20260806 02:46:09.218778 9461 consensus_queue.cc:260] T 97ac65438b414347b0861dee921602a4 P cea96d710f5c430db2f412a731bd1bee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea96d710f5c430db2f412a731bd1bee" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 42461 } } peers { permanent_uuid: "f4874064dc7841d2b5578c350facdba9" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 34719 } } peers { permanent_uuid: "04982070d1fd4a2db220580b0b57043c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 33381 } }
I20260806 02:46:09.219861 9461 ts_tablet_manager.cc:1428] T 97ac65438b414347b0861dee921602a4 P cea96d710f5c430db2f412a731bd1bee: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 02:46:09.223839 9070 heartbeater.cc:503] Master 127.1.210.124:44901 was elected leader, sending a full tablet report...
I20260806 02:46:09.227736 9463 tablet_bootstrap.cc:492] T 97ac65438b414347b0861dee921602a4 P 04982070d1fd4a2db220580b0b57043c: No bootstrap required, opened a new log
I20260806 02:46:09.227978 9463 ts_tablet_manager.cc:1397] T 97ac65438b414347b0861dee921602a4 P 04982070d1fd4a2db220580b0b57043c: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260806 02:46:09.229666 9463 raft_consensus.cc:348] T 97ac65438b414347b0861dee921602a4 P 04982070d1fd4a2db220580b0b57043c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea96d710f5c430db2f412a731bd1bee" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 42461 } } peers { permanent_uuid: "f4874064dc7841d2b5578c350facdba9" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 34719 } } peers { permanent_uuid: "04982070d1fd4a2db220580b0b57043c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 33381 } }
I20260806 02:46:09.230051 9463 raft_consensus.cc:374] T 97ac65438b414347b0861dee921602a4 P 04982070d1fd4a2db220580b0b57043c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:09.230177 9463 raft_consensus.cc:729] T 97ac65438b414347b0861dee921602a4 P 04982070d1fd4a2db220580b0b57043c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04982070d1fd4a2db220580b0b57043c, State: Initialized, Role: FOLLOWER
I20260806 02:46:09.230574 9463 consensus_queue.cc:260] T 97ac65438b414347b0861dee921602a4 P 04982070d1fd4a2db220580b0b57043c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea96d710f5c430db2f412a731bd1bee" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 42461 } } peers { permanent_uuid: "f4874064dc7841d2b5578c350facdba9" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 34719 } } peers { permanent_uuid: "04982070d1fd4a2db220580b0b57043c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 33381 } }
I20260806 02:46:09.231896 9466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.220571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.1.210.65:33517)
0806 02:46:09.231323 (+ 10752us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:09.231486 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:41363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10619,"thread_start_us":10462,"threads_started":1}
I20260806 02:46:09.232620 9467 tablet_bootstrap.cc:492] T 97ac65438b414347b0861dee921602a4 P f4874064dc7841d2b5578c350facdba9: Bootstrap starting.
I20260806 02:46:09.233271 9463 ts_tablet_manager.cc:1428] T 97ac65438b414347b0861dee921602a4 P 04982070d1fd4a2db220580b0b57043c: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260806 02:46:09.233944 9466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.233310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.1.210.65:44949)
0806 02:46:09.233677 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:09.233778 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:41363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242}
I20260806 02:46:09.234766 9467 tablet_bootstrap.cc:654] T 97ac65438b414347b0861dee921602a4 P f4874064dc7841d2b5578c350facdba9: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:09.236168 9361 heartbeater.cc:503] Master 127.1.210.124:44901 was elected leader, sending a full tablet report...
I20260806 02:46:09.239159 9467 tablet_bootstrap.cc:492] T 97ac65438b414347b0861dee921602a4 P f4874064dc7841d2b5578c350facdba9: No bootstrap required, opened a new log
I20260806 02:46:09.239380 9467 ts_tablet_manager.cc:1397] T 97ac65438b414347b0861dee921602a4 P f4874064dc7841d2b5578c350facdba9: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260806 02:46:09.240527 9467 raft_consensus.cc:348] T 97ac65438b414347b0861dee921602a4 P f4874064dc7841d2b5578c350facdba9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea96d710f5c430db2f412a731bd1bee" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 42461 } } peers { permanent_uuid: "f4874064dc7841d2b5578c350facdba9" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 34719 } } peers { permanent_uuid: "04982070d1fd4a2db220580b0b57043c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 33381 } }
I20260806 02:46:09.240862 9467 raft_consensus.cc:374] T 97ac65438b414347b0861dee921602a4 P f4874064dc7841d2b5578c350facdba9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:09.240988 9467 raft_consensus.cc:729] T 97ac65438b414347b0861dee921602a4 P f4874064dc7841d2b5578c350facdba9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4874064dc7841d2b5578c350facdba9, State: Initialized, Role: FOLLOWER
I20260806 02:46:09.241319 9467 consensus_queue.cc:260] T 97ac65438b414347b0861dee921602a4 P f4874064dc7841d2b5578c350facdba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea96d710f5c430db2f412a731bd1bee" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 42461 } } peers { permanent_uuid: "f4874064dc7841d2b5578c350facdba9" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 34719 } } peers { permanent_uuid: "04982070d1fd4a2db220580b0b57043c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 33381 } }
I20260806 02:46:09.242518 9467 ts_tablet_manager.cc:1428] T 97ac65438b414347b0861dee921602a4 P f4874064dc7841d2b5578c350facdba9: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
I20260806 02:46:09.244115 9472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.243017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.1.210.66:51803)
0806 02:46:09.243674 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:09.243850 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:41363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":450,"threads_started":1}
I20260806 02:46:09.245146 9219 heartbeater.cc:503] Master 127.1.210.124:44901 was elected leader, sending a full tablet report...
I20260806 02:46:09.246047 9472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.245434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.1.210.66:51015)
0806 02:46:09.245818 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:09.245902 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:41363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284}
I20260806 02:46:09.246059 9469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.233883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.1.210.67:53205)
0806 02:46:09.245601 (+ 11718us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:09.245771 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:41363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11552,"threads_started":1}
I20260806 02:46:09.247377 9469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.246893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.1.210.67:48089)
0806 02:46:09.247085 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:09.247194 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:41363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 02:46:09.251924 9468 raft_consensus.cc:482] T 97ac65438b414347b0861dee921602a4 P 04982070d1fd4a2db220580b0b57043c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:46:09.252220 9468 raft_consensus.cc:504] T 97ac65438b414347b0861dee921602a4 P 04982070d1fd4a2db220580b0b57043c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea96d710f5c430db2f412a731bd1bee" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 42461 } } peers { permanent_uuid: "f4874064dc7841d2b5578c350facdba9" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 34719 } } peers { permanent_uuid: "04982070d1fd4a2db220580b0b57043c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 33381 } }
I20260806 02:46:09.253872 9468 leader_election.cc:288] T 97ac65438b414347b0861dee921602a4 P 04982070d1fd4a2db220580b0b57043c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea96d710f5c430db2f412a731bd1bee (127.1.210.65:42461), f4874064dc7841d2b5578c350facdba9 (127.1.210.66:34719)
I20260806 02:46:09.260437 9460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.254303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:37151 (local address 127.1.210.66:34719)
0806 02:46:09.254496 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:09.254502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:09.255159 (+ 657us) server_negotiation.cc:407] Connection header received
0806 02:46:09.255351 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:09.255356 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:09.255441 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:09.255544 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:46:09.256141 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.256915 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.258751 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.259517 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.259679 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:09.259757 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:09.260076 (+ 319us) server_negotiation.cc:299] Negotiation successful
0806 02:46:09.260211 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 02:46:09.260464 9457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.254312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:33707 (local address 127.1.210.65:42461)
0806 02:46:09.254500 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:09.254506 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:09.254525 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:46:09.254819 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:09.254827 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:09.254925 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:09.255033 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:46:09.255688 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.256625 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.258457 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.259413 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.259549 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:09.259642 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:09.260029 (+ 387us) server_negotiation.cc:299] Negotiation successful
0806 02:46:09.260170 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 02:46:09.260473 9469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.254125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:42461 (local address 127.1.210.67:33707)
0806 02:46:09.254379 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:09.254402 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:09.254619 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:09.255089 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:09.255093 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:09.255201 (+ 108us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:46:09.255494 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:09.255544 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.256767 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.256772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:09.258292 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:09.258305 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.259608 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.259614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:09.259847 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:09.259893 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 02:46:09.259969 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 02:46:09.260148 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":6,"spinlock_wait_cycles":1408}
I20260806 02:46:09.261133 9473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.254186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:34719 (local address 127.1.210.67:37151)
0806 02:46:09.255021 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:09.255053 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:09.255192 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:09.255562 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:09.255565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:09.255584 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:46:09.255926 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:09.255938 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.257094 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.257098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:09.258550 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:09.258564 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.259729 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.259737 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:09.259892 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:09.259915 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 02:46:09.260034 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 02:46:09.260165 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"mutex_wait_us":41,"spinlock_wait_cycles":3456,"thread_start_us":523,"threads_started":1}
I20260806 02:46:09.261354 9024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ac65438b414347b0861dee921602a4" candidate_uuid: "04982070d1fd4a2db220580b0b57043c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea96d710f5c430db2f412a731bd1bee" is_pre_election: true
I20260806 02:46:09.261866 9174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ac65438b414347b0861dee921602a4" candidate_uuid: "04982070d1fd4a2db220580b0b57043c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4874064dc7841d2b5578c350facdba9" is_pre_election: true
I20260806 02:46:09.261888 9024 raft_consensus.cc:2454] T 97ac65438b414347b0861dee921602a4 P cea96d710f5c430db2f412a731bd1bee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04982070d1fd4a2db220580b0b57043c in term 0.
I20260806 02:46:09.262109 9174 raft_consensus.cc:2454] T 97ac65438b414347b0861dee921602a4 P f4874064dc7841d2b5578c350facdba9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04982070d1fd4a2db220580b0b57043c in term 0.
I20260806 02:46:09.262537 9248 leader_election.cc:302] T 97ac65438b414347b0861dee921602a4 P 04982070d1fd4a2db220580b0b57043c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04982070d1fd4a2db220580b0b57043c, cea96d710f5c430db2f412a731bd1bee; no voters:
I20260806 02:46:09.263022 9468 raft_consensus.cc:2781] T 97ac65438b414347b0861dee921602a4 P 04982070d1fd4a2db220580b0b57043c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:46:09.263209 9468 raft_consensus.cc:482] T 97ac65438b414347b0861dee921602a4 P 04982070d1fd4a2db220580b0b57043c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:46:09.263320 9468 raft_consensus.cc:3037] T 97ac65438b414347b0861dee921602a4 P 04982070d1fd4a2db220580b0b57043c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:09.265517 9468 raft_consensus.cc:504] T 97ac65438b414347b0861dee921602a4 P 04982070d1fd4a2db220580b0b57043c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea96d710f5c430db2f412a731bd1bee" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 42461 } } peers { permanent_uuid: "f4874064dc7841d2b5578c350facdba9" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 34719 } } peers { permanent_uuid: "04982070d1fd4a2db220580b0b57043c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 33381 } }
I20260806 02:46:09.266227 9468 leader_election.cc:288] T 97ac65438b414347b0861dee921602a4 P 04982070d1fd4a2db220580b0b57043c [CANDIDATE]: Term 1 election: Requested vote from peers cea96d710f5c430db2f412a731bd1bee (127.1.210.65:42461), f4874064dc7841d2b5578c350facdba9 (127.1.210.66:34719)
I20260806 02:46:09.266846 9024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ac65438b414347b0861dee921602a4" candidate_uuid: "04982070d1fd4a2db220580b0b57043c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea96d710f5c430db2f412a731bd1bee"
I20260806 02:46:09.266919 9174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ac65438b414347b0861dee921602a4" candidate_uuid: "04982070d1fd4a2db220580b0b57043c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4874064dc7841d2b5578c350facdba9"
I20260806 02:46:09.267110 9024 raft_consensus.cc:3037] T 97ac65438b414347b0861dee921602a4 P cea96d710f5c430db2f412a731bd1bee [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:09.267217 9174 raft_consensus.cc:3037] T 97ac65438b414347b0861dee921602a4 P f4874064dc7841d2b5578c350facdba9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:09.269354 9174 raft_consensus.cc:2454] T 97ac65438b414347b0861dee921602a4 P f4874064dc7841d2b5578c350facdba9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04982070d1fd4a2db220580b0b57043c in term 1.
I20260806 02:46:09.269382 9024 raft_consensus.cc:2454] T 97ac65438b414347b0861dee921602a4 P cea96d710f5c430db2f412a731bd1bee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04982070d1fd4a2db220580b0b57043c in term 1.
I20260806 02:46:09.269833 9250 leader_election.cc:302] T 97ac65438b414347b0861dee921602a4 P 04982070d1fd4a2db220580b0b57043c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04982070d1fd4a2db220580b0b57043c, f4874064dc7841d2b5578c350facdba9; no voters:
I20260806 02:46:09.270228 9468 raft_consensus.cc:2781] T 97ac65438b414347b0861dee921602a4 P 04982070d1fd4a2db220580b0b57043c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:46:09.270834 9468 raft_consensus.cc:686] T 97ac65438b414347b0861dee921602a4 P 04982070d1fd4a2db220580b0b57043c [term 1 LEADER]: Becoming Leader. State: Replica: 04982070d1fd4a2db220580b0b57043c, State: Running, Role: LEADER
I20260806 02:46:09.271302 9468 consensus_queue.cc:237] T 97ac65438b414347b0861dee921602a4 P 04982070d1fd4a2db220580b0b57043c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea96d710f5c430db2f412a731bd1bee" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 42461 } } peers { permanent_uuid: "f4874064dc7841d2b5578c350facdba9" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 34719 } } peers { permanent_uuid: "04982070d1fd4a2db220580b0b57043c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 33381 } }
I20260806 02:46:09.272020 9473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.271602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.1.210.67:54153)
0806 02:46:09.271768 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:09.271886 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:41363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 02:46:09.275342 8890 catalog_manager.cc:5495] T 97ac65438b414347b0861dee921602a4 P 04982070d1fd4a2db220580b0b57043c reported cstate change: term changed from 0 to 1, leader changed from <none> to 04982070d1fd4a2db220580b0b57043c (127.1.210.67). New cstate: current_term: 1 leader_uuid: "04982070d1fd4a2db220580b0b57043c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea96d710f5c430db2f412a731bd1bee" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 42461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4874064dc7841d2b5578c350facdba9" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 34719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04982070d1fd4a2db220580b0b57043c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 33381 } health_report { overall_health: HEALTHY } } }
I20260806 02:46:09.305729 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.298602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36818 (local address 127.1.210.67:33381)
0806 02:46:09.298908 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:09.298919 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:09.298937 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:46:09.299039 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:09.299043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:09.299136 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:09.299270 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:46:09.299746 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.301139 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.303185 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.303719 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.303877 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:09.304408 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:46:09.305176 (+ 768us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:46:09.305298 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:09.305431 (+ 133us) server_negotiation.cc:299] Negotiation successful
0806 02:46:09.305511 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260806 02:46:09.310304 9174 raft_consensus.cc:1261] T 97ac65438b414347b0861dee921602a4 P f4874064dc7841d2b5578c350facdba9 [term 1 FOLLOWER]: Refusing update from remote peer 04982070d1fd4a2db220580b0b57043c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:46:09.310804 9024 raft_consensus.cc:1261] T 97ac65438b414347b0861dee921602a4 P cea96d710f5c430db2f412a731bd1bee [term 1 FOLLOWER]: Refusing update from remote peer 04982070d1fd4a2db220580b0b57043c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:46:09.311097 9474 consensus_queue.cc:1035] T 97ac65438b414347b0861dee921602a4 P 04982070d1fd4a2db220580b0b57043c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4874064dc7841d2b5578c350facdba9" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 34719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:46:09.311607 9468 consensus_queue.cc:1035] T 97ac65438b414347b0861dee921602a4 P 04982070d1fd4a2db220580b0b57043c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea96d710f5c430db2f412a731bd1bee" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 42461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:46:09.313970 9472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.313482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.1.210.66:52213)
0806 02:46:09.313665 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:09.313773 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:41363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260806 02:46:09.315083 9466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.314589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.1.210.65:42549)
0806 02:46:09.314756 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:09.314892 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:41363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260806 02:46:09.319146 9478 mvcc.cc:205] Tried to move back new op lower bound from 7315391976687505408 to 7315391976539607040. Current Snapshot: MvccSnapshot[applied={T|T < 7315391976687505408}]
I20260806 02:46:09.321211 9473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.320020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.1.210.67:55159)
0806 02:46:09.320823 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:09.320963 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:41363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":631}
I20260806 02:46:09.324627 9466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.320626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.1.210.65:39883)
0806 02:46:09.324318 (+ 3692us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:09.324425 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:41363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3572}
I20260806 02:46:09.331909 9473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.331246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.1.210.67:41699)
0806 02:46:09.331584 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:09.331726 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:41363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226}
I20260806 02:46:09.334136 9472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.333749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.1.210.66:59615)
0806 02:46:09.333910 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:09.333998 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:41363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260806 02:46:09.337596 8890 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:46:09.337968 8890 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:46:09.344224 8890 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9cd0128d2476470cb51e77201cfb727a: Sending DeleteTablet for 3 replicas of tablet 97ac65438b414347b0861dee921602a4
I20260806 02:46:09.345394 9004 tablet_service.cc:1514] Processing DeleteTablet for tablet 97ac65438b414347b0861dee921602a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:46:09 UTC) from {username='oryx', principal='oryx/127.1.210.124@KRBTEST.COM'} at 127.0.0.1:52220
I20260806 02:46:09.345470 9296 tablet_service.cc:1514] Processing DeleteTablet for tablet 97ac65438b414347b0861dee921602a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:46:09 UTC) from {username='oryx', principal='oryx/127.1.210.124@KRBTEST.COM'} at 127.0.0.1:36814
I20260806 02:46:09.345548 9154 tablet_service.cc:1514] Processing DeleteTablet for tablet 97ac65438b414347b0861dee921602a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:46:09 UTC) from {username='oryx', principal='oryx/127.1.210.124@KRBTEST.COM'} at 127.0.0.1:51488
I20260806 02:46:09.351722 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 8863
I20260806 02:46:09.355587 9489 tablet_replica.cc:330] T 97ac65438b414347b0861dee921602a4 P f4874064dc7841d2b5578c350facdba9: stopping tablet replica
I20260806 02:46:09.355834 9489 raft_consensus.cc:2229] T 97ac65438b414347b0861dee921602a4 P f4874064dc7841d2b5578c350facdba9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:46:09.356204 9489 raft_consensus.cc:2258] T 97ac65438b414347b0861dee921602a4 P f4874064dc7841d2b5578c350facdba9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:46:09.359017 9489 ts_tablet_manager.cc:1905] T 97ac65438b414347b0861dee921602a4 P f4874064dc7841d2b5578c350facdba9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:46:09.359632 9487 tablet_replica.cc:330] T 97ac65438b414347b0861dee921602a4 P cea96d710f5c430db2f412a731bd1bee: stopping tablet replica
I20260806 02:46:09.359931 9487 raft_consensus.cc:2229] T 97ac65438b414347b0861dee921602a4 P cea96d710f5c430db2f412a731bd1bee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:46:09.360357 9487 raft_consensus.cc:2258] T 97ac65438b414347b0861dee921602a4 P cea96d710f5c430db2f412a731bd1bee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:46:09.363061 9487 ts_tablet_manager.cc:1905] T 97ac65438b414347b0861dee921602a4 P cea96d710f5c430db2f412a731bd1bee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:46:09.363314 9488 tablet_replica.cc:330] T 97ac65438b414347b0861dee921602a4 P 04982070d1fd4a2db220580b0b57043c: stopping tablet replica
I20260806 02:46:09.363566 9488 raft_consensus.cc:2229] T 97ac65438b414347b0861dee921602a4 P 04982070d1fd4a2db220580b0b57043c [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:46:09.364288 9488 raft_consensus.cc:2258] T 97ac65438b414347b0861dee921602a4 P 04982070d1fd4a2db220580b0b57043c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:46:09.367604 9488 ts_tablet_manager.cc:1905] T 97ac65438b414347b0861dee921602a4 P 04982070d1fd4a2db220580b0b57043c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:46:09.368116 9489 ts_tablet_manager.cc:1918] T 97ac65438b414347b0861dee921602a4 P f4874064dc7841d2b5578c350facdba9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:46:09.368270 9489 log.cc:1196] T 97ac65438b414347b0861dee921602a4 P f4874064dc7841d2b5578c350facdba9: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-1/wal/wals/97ac65438b414347b0861dee921602a4
I20260806 02:46:09.368722 9489 ts_tablet_manager.cc:1939] T 97ac65438b414347b0861dee921602a4 P f4874064dc7841d2b5578c350facdba9: Deleting consensus metadata
I20260806 02:46:09.370100 8877 catalog_manager.cc:4841] TS f4874064dc7841d2b5578c350facdba9 (127.1.210.66:34719): tablet 97ac65438b414347b0861dee921602a4 (table test-table [id=f7784dab20a745e0aee2464a06077d55]) successfully deleted
I20260806 02:46:09.374346 9487 ts_tablet_manager.cc:1918] T 97ac65438b414347b0861dee921602a4 P cea96d710f5c430db2f412a731bd1bee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:46:09.374504 9487 log.cc:1196] T 97ac65438b414347b0861dee921602a4 P cea96d710f5c430db2f412a731bd1bee: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-0/wal/wals/97ac65438b414347b0861dee921602a4
I20260806 02:46:09.375176 9487 ts_tablet_manager.cc:1939] T 97ac65438b414347b0861dee921602a4 P cea96d710f5c430db2f412a731bd1bee: Deleting consensus metadata
I20260806 02:46:09.375555 9488 ts_tablet_manager.cc:1918] T 97ac65438b414347b0861dee921602a4 P 04982070d1fd4a2db220580b0b57043c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:46:09.375756 9488 log.cc:1196] T 97ac65438b414347b0861dee921602a4 P 04982070d1fd4a2db220580b0b57043c: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984304136196-1865-0/minicluster-data/ts-2/wal/wals/97ac65438b414347b0861dee921602a4
I20260806 02:46:09.376314 9488 ts_tablet_manager.cc:1939] T 97ac65438b414347b0861dee921602a4 P 04982070d1fd4a2db220580b0b57043c: Deleting consensus metadata
I20260806 02:46:09.376523 8875 catalog_manager.cc:4841] TS cea96d710f5c430db2f412a731bd1bee (127.1.210.65:42461): tablet 97ac65438b414347b0861dee921602a4 (table test-table [id=f7784dab20a745e0aee2464a06077d55]) successfully deleted
I20260806 02:46:09.377628 8876 catalog_manager.cc:4841] TS 04982070d1fd4a2db220580b0b57043c (127.1.210.67:33381): tablet 97ac65438b414347b0861dee921602a4 (table test-table [id=f7784dab20a745e0aee2464a06077d55]) successfully deleted
I20260806 02:46:09.381274 9457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.353100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52232 (local address 127.1.210.65:42461)
0806 02:46:09.353397 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:09.353407 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:09.355926 (+ 2519us) server_negotiation.cc:407] Connection header received
0806 02:46:09.356444 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:09.356452 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:09.356585 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:09.356782 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:09.361088 (+ 4306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.362276 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.369240 (+ 6964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.369863 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.369967 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:09.373912 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:09.373935 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:09.373957 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:09.374016 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:09.378171 (+ 4155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:09.379059 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:09.379067 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:09.379069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:09.379206 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:09.379626 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:09.379638 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:09.379641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:09.380034 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:09.380355 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:09.380672 (+ 317us) server_negotiation.cc:299] Negotiation successful
0806 02:46:09.380919 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260806 02:46:09.382004 9065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:46:09.508603 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 8863
I20260806 02:46:09.534570 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 9089
I20260806 02:46:09.552309 9460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.535722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51496 (local address 127.1.210.66:34719)
0806 02:46:09.535945 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:09.535953 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:09.536009 (+ 56us) server_negotiation.cc:407] Connection header received
0806 02:46:09.536190 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:09.536196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:09.536306 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:09.536410 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:09.537921 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.539163 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.540458 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.541078 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.541212 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:09.544472 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:09.544495 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:09.544516 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:09.544590 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:09.548868 (+ 4278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:09.549742 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:09.549755 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:09.549759 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:09.549926 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:09.550447 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:09.550453 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:09.550457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:09.550990 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:09.551308 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:09.551658 (+ 350us) server_negotiation.cc:299] Negotiation successful
0806 02:46:09.551924 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 02:46:09.553165 9214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:46:09.653775 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 9089
I20260806 02:46:09.672901 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 9232
I20260806 02:46:09.691926 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.673729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36828 (local address 127.1.210.67:33381)
0806 02:46:09.673911 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:09.673917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:09.673990 (+ 73us) server_negotiation.cc:407] Connection header received
0806 02:46:09.674213 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:09.674221 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:09.674308 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:09.674438 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:09.676562 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.677490 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.679672 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.680204 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.680339 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:09.684290 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:09.684307 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:09.684324 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:09.684368 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:09.688300 (+ 3932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:09.689336 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:09.689344 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:09.689346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:09.689522 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:09.690133 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:09.690138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:09.690141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:09.690578 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:09.690968 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:09.691323 (+ 355us) server_negotiation.cc:299] Negotiation successful
0806 02:46:09.691522 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 02:46:09.692777 9356 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:46:09.702421 9441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.701974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.0.0.1:41930)
0806 02:46:09.702173 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:09.702277 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:41363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 02:46:09.703953 9441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.703572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41363 (local address 127.0.0.1:41932)
0806 02:46:09.703752 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:09.703833 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:41363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 02:46:09.833441 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 9232
I20260806 02:46:09.856788 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 8557
I20260806 02:46:09.858206 8732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:46:09.983682 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 8557
I20260806 02:46:10.011620 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 8653
I20260806 02:46:10.012913 8921 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:46:10.139374 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 8653
2026-08-06T02:46:10Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10697 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:46:10 dist-test-slave-8tdl krb5kdc[9499](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:46:10 dist-test-slave-8tdl krb5kdc[9499](info): set up 2 sockets
Aug 06 02:46:10 dist-test-slave-8tdl krb5kdc[9499](info): commencing operation
krb5kdc: starting...
W20260806 02:46:11.324659 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.137s user 0.001s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:46:11 dist-test-slave-8tdl krb5kdc[9499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:46:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:46:11Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.126.keytab.
Entry for principal oryx/127.1.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.126.keytab.
I20260806 02:46:11.588465 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:36511
--webserver_interface=127.1.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.126.keytab
--principal=oryx/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:37149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:36511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:46:11.943627 9512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:11.943956 9512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:11.944125 9512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:11.951653 9512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:46:11.951783 9512 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 02:46:11.951862 9512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:11.951963 9512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:11.952013 9512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:46:11.952153 9512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:46:11.962759 9512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:37149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:36511
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.126.keytab
--principal=oryx/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:36511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:46:11.964272 9512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:11.966123 9512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:11.981160 9517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:11.981235 9518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:11.981187 9520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:11.984522 9512 server_base.cc:1034] running on GCE node
I20260806 02:46:11.985455 9512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:11.987174 9512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:11.988461 9512 hybrid_clock.cc:648] HybridClock initialized: now 1785984371988389 us; error 55 us; skew 500 ppm
I20260806 02:46:11.988852 9512 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:46:11 dist-test-slave-8tdl krb5kdc[9499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984371, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:11.992424 9512 init.cc:377] Logged in from keytab as oryx/127.1.210.126@KRBTEST.COM (short username oryx)
I20260806 02:46:11.994199 9512 webserver.cc:460] Webserver started at http://127.1.210.126:46373/ using document root <none> and password file <none>
I20260806 02:46:11.995347 9512 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:11.995496 9512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:11.995941 9512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:11.999159 9512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "1e49507f75394515a4d7645a0f7848c1"
format_stamp: "Formatted at 2026-08-06 02:46:11 on dist-test-slave-8tdl"
server_key: "b023031e0b75eda70bdcee3b63764763"
server_key_iv: "5ddd37877b3cb680b6be07b419371da9"
server_key_version: "encryptionkey@0"
I20260806 02:46:12.000041 9512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "1e49507f75394515a4d7645a0f7848c1"
format_stamp: "Formatted at 2026-08-06 02:46:11 on dist-test-slave-8tdl"
server_key: "b023031e0b75eda70bdcee3b63764763"
server_key_iv: "5ddd37877b3cb680b6be07b419371da9"
server_key_version: "encryptionkey@0"
I20260806 02:46:12.006220 9512 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260806 02:46:12.011636 9527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:12.013396 9512 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:46:12.013636 9512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "1e49507f75394515a4d7645a0f7848c1"
format_stamp: "Formatted at 2026-08-06 02:46:11 on dist-test-slave-8tdl"
server_key: "b023031e0b75eda70bdcee3b63764763"
server_key_iv: "5ddd37877b3cb680b6be07b419371da9"
server_key_version: "encryptionkey@0"
I20260806 02:46:12.013892 9512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:12.050019 9512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:12.055301 9512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:12.055644 9512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:12.081779 9512 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:36511
I20260806 02:46:12.081782 9589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:36511 every 8 connection(s)
I20260806 02:46:12.083514 9512 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:46:12.085758 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 9512
I20260806 02:46:12.085986 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:46:12.086938 1865 external_mini_cluster.cc:1428] Setting key 9a092934215fc78d21f6c411495c6d49
I20260806 02:46:12.090586 9590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 02:46:12 dist-test-slave-8tdl krb5kdc[9499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM
I20260806 02:46:12.102934 9590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e49507f75394515a4d7645a0f7848c1: Bootstrap starting.
I20260806 02:46:12.106786 9590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e49507f75394515a4d7645a0f7848c1: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:12.107919 9590 log.cc:824] T 00000000000000000000000000000000 P 1e49507f75394515a4d7645a0f7848c1: Log is configured to *not* fsync() on all Append() calls
I20260806 02:46:12.111713 9590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e49507f75394515a4d7645a0f7848c1: No bootstrap required, opened a new log
I20260806 02:46:12.112062 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:12.089370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34332 (local address 127.1.210.126:36511)
0806 02:46:12.090346 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:12.090361 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:12.090414 (+ 53us) server_negotiation.cc:407] Connection header received
0806 02:46:12.092312 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:12.092337 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:12.092776 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:12.093223 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:12.094999 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:12.096463 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:12.097791 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:12.098532 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:12.098776 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:12.103442 (+ 4666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:12.103452 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:12.103469 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:12.103516 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:12.107008 (+ 3492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:12.107815 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:12.107823 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:12.107829 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:12.108009 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:12.108515 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:12.108523 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:12.108539 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:12.109175 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:12.109524 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:12.109941 (+ 417us) server_negotiation.cc:299] Negotiation successful
0806 02:46:12.110405 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":733,"thread_start_us":433,"threads_started":1}
I20260806 02:46:12.120704 9590 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e49507f75394515a4d7645a0f7848c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e49507f75394515a4d7645a0f7848c1" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 36511 } }
I20260806 02:46:12.121122 9590 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e49507f75394515a4d7645a0f7848c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:12.121260 9590 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e49507f75394515a4d7645a0f7848c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e49507f75394515a4d7645a0f7848c1, State: Initialized, Role: FOLLOWER
I20260806 02:46:12.121863 9590 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e49507f75394515a4d7645a0f7848c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e49507f75394515a4d7645a0f7848c1" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 36511 } }
I20260806 02:46:12.122145 9590 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e49507f75394515a4d7645a0f7848c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:46:12.122311 9590 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e49507f75394515a4d7645a0f7848c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:46:12.122512 9590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e49507f75394515a4d7645a0f7848c1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:12.125181 9590 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e49507f75394515a4d7645a0f7848c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e49507f75394515a4d7645a0f7848c1" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 36511 } }
I20260806 02:46:12.125752 9590 leader_election.cc:302] T 00000000000000000000000000000000 P 1e49507f75394515a4d7645a0f7848c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e49507f75394515a4d7645a0f7848c1; no voters:
I20260806 02:46:12.126546 9590 leader_election.cc:288] T 00000000000000000000000000000000 P 1e49507f75394515a4d7645a0f7848c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:46:12.126708 9595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e49507f75394515a4d7645a0f7848c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:46:12.127532 9595 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e49507f75394515a4d7645a0f7848c1 [term 1 LEADER]: Becoming Leader. State: Replica: 1e49507f75394515a4d7645a0f7848c1, State: Running, Role: LEADER
I20260806 02:46:12.128064 9595 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e49507f75394515a4d7645a0f7848c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e49507f75394515a4d7645a0f7848c1" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 36511 } }
I20260806 02:46:12.128809 9590 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e49507f75394515a4d7645a0f7848c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:46:12.131510 9597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e49507f75394515a4d7645a0f7848c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e49507f75394515a4d7645a0f7848c1. Latest consensus state: current_term: 1 leader_uuid: "1e49507f75394515a4d7645a0f7848c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e49507f75394515a4d7645a0f7848c1" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 36511 } } }
I20260806 02:46:12.131987 9597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e49507f75394515a4d7645a0f7848c1 [sys.catalog]: This master's current role is: LEADER
I20260806 02:46:12.132829 9596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e49507f75394515a4d7645a0f7848c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e49507f75394515a4d7645a0f7848c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e49507f75394515a4d7645a0f7848c1" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 36511 } } }
I20260806 02:46:12.138612 9596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e49507f75394515a4d7645a0f7848c1 [sys.catalog]: This master's current role is: LEADER
I20260806 02:46:12.139845 9605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:46:12.144663 9605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:46:12.154371 9605 catalog_manager.cc:1347] Generated new cluster ID: 33299aa1d13044a6971b551c8edcfaf5
I20260806 02:46:12.154517 9605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:46:12.163815 9605 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:46:12.165175 9605 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:46:12.173342 9605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e49507f75394515a4d7645a0f7848c1: Generated new TSK 0
I20260806 02:46:12.174284 9605 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.65.keytab.
Entry for principal oryx/127.1.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.65.keytab.
I20260806 02:46:12.248286 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.65:0
--local_ip_for_outbound_sockets=127.1.210.65
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:36511
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.65.keytab
--principal=oryx/127.1.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:37149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:46:12.606488 9617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:12.606896 9617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:12.607105 9617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:12.615041 9617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:12.615163 9617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:12.615329 9617 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:46:12.615471 9617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.65
I20260806 02:46:12.626032 9617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:37149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.65.keytab
--principal=oryx/127.1.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.210.126:36511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.65
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:46:12.627731 9617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:12.629390 9617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:12.643721 9622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:12.643752 9623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:12.643781 9625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:12.644830 9617 server_base.cc:1034] running on GCE node
I20260806 02:46:12.645543 9617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:12.646624 9617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:12.647922 9617 hybrid_clock.cc:648] HybridClock initialized: now 1785984372647845 us; error 45 us; skew 500 ppm
I20260806 02:46:12.648285 9617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:46:12 dist-test-slave-8tdl krb5kdc[9499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984372, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:12.651832 9617 init.cc:377] Logged in from keytab as oryx/127.1.210.65@KRBTEST.COM (short username oryx)
I20260806 02:46:12.653486 9617 webserver.cc:460] Webserver started at http://127.1.210.65:38639/ using document root <none> and password file <none>
I20260806 02:46:12.654172 9617 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:12.654260 9617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:12.654536 9617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:12.657608 9617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-0/data/instance:
uuid: "7d735936b19c4d7aa4b8db79c7694f1d"
format_stamp: "Formatted at 2026-08-06 02:46:12 on dist-test-slave-8tdl"
server_key: "7619fa3f22f788819c9879614ef5cde4"
server_key_iv: "6d60d86aa1d8264857d14ee56903aaa6"
server_key_version: "encryptionkey@0"
I20260806 02:46:12.658453 9617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance:
uuid: "7d735936b19c4d7aa4b8db79c7694f1d"
format_stamp: "Formatted at 2026-08-06 02:46:12 on dist-test-slave-8tdl"
server_key: "7619fa3f22f788819c9879614ef5cde4"
server_key_iv: "6d60d86aa1d8264857d14ee56903aaa6"
server_key_version: "encryptionkey@0"
I20260806 02:46:12.663939 9617 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 02:46:12.667644 9632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:12.669297 9617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:46:12.669533 9617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-0/wal
uuid: "7d735936b19c4d7aa4b8db79c7694f1d"
format_stamp: "Formatted at 2026-08-06 02:46:12 on dist-test-slave-8tdl"
server_key: "7619fa3f22f788819c9879614ef5cde4"
server_key_iv: "6d60d86aa1d8264857d14ee56903aaa6"
server_key_version: "encryptionkey@0"
I20260806 02:46:12.669759 9617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:12.699679 9617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:12.704690 9617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:12.705003 9617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:12.707991 9617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:12.708114 9617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:12.708298 9617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:12.708441 9617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Aug 06 02:46:12 dist-test-slave-8tdl krb5kdc[9499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984372, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM
I20260806 02:46:12.738375 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:12.709528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:36511 (local address 127.1.210.65:36521)
0806 02:46:12.710465 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:12.710522 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:12.712499 (+ 1977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:12.713330 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:12.713349 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:12.714389 (+ 1040us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:12.715318 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:12.715344 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:12.717269 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:12.717278 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:12.719212 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:12.719237 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:12.720598 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:12.720609 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:12.720911 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:12.721744 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:12.721791 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:12.726539 (+ 4748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:12.731165 (+ 4626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:12.731176 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:12.731197 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:12.732038 (+ 841us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:12.732686 (+ 648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:12.732695 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:12.732702 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:12.733210 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:12.734152 (+ 942us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:12.734164 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:12.734857 (+ 693us) client_negotiation.cc:769] Sending connection context
0806 02:46:12.735297 (+ 440us) client_negotiation.cc:240] Negotiation successful
0806 02:46:12.736107 (+ 810us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":396,"threads_started":1}
I20260806 02:46:12.740051 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:12.708659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:36521 (local address 127.1.210.126:36511)
0806 02:46:12.709510 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:12.709517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:12.710695 (+ 1178us) server_negotiation.cc:407] Connection header received
0806 02:46:12.712804 (+ 2109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:12.712810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:12.712901 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:12.713043 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:12.715663 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:12.717004 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:12.719437 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:12.720270 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:12.720579 (+ 309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:12.726946 (+ 6367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:12.726956 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:12.726961 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:12.727019 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:12.730967 (+ 3948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:12.732265 (+ 1298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:12.732272 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:12.732276 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:12.732465 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:12.733408 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:12.733417 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:12.733423 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:12.733862 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:12.734056 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:12.739205 (+ 5149us) server_negotiation.cc:299] Negotiation successful
0806 02:46:12.739540 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":593,"threads_started":1}
I20260806 02:46:12.757302 9617 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.65:33721
I20260806 02:46:12.757323 9754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.65:33721 every 8 connection(s)
I20260806 02:46:12.759043 9617 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
I20260806 02:46:12.761433 9755 heartbeater.cc:347] Connected to a master server at 127.1.210.126:36511
I20260806 02:46:12.761723 9755 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:12.762641 9755 heartbeater.cc:511] Master 127.1.210.126:36511 requested a full tablet report, sending...
I20260806 02:46:12.764284 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:12.746667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:53563 (local address 127.1.210.126:36511)
0806 02:46:12.746910 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:12.746919 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:12.747796 (+ 877us) server_negotiation.cc:407] Connection header received
0806 02:46:12.747935 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:12.747944 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:12.748078 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:12.748229 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:12.749273 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:12.750835 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:12.753818 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:12.754721 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:12.754889 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:12.758074 (+ 3185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:12.758079 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:12.758083 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:12.758122 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:12.761306 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:12.762183 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:12.762187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:12.762189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:12.762324 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:12.762775 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:12.762780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:12.762782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:12.763133 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:12.763242 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:12.763795 (+ 553us) server_negotiation.cc:299] Negotiation successful
0806 02:46:12.764028 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 02:46:12.764489 9737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:12.746810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:36511 (local address 127.1.210.65:53563)
0806 02:46:12.747649 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:12.747669 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:12.747814 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:12.748290 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:12.748294 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:12.748781 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:12.749099 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:12.749110 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:12.751100 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:12.751109 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:12.753547 (+ 2438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:12.753575 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:12.755260 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:12.755267 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:12.755424 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:12.756346 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:12.756385 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:12.757814 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:12.761460 (+ 3646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:12.761467 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:12.761471 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:12.762063 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:12.762458 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:12.762462 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:12.762466 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:12.762643 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:12.763282 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:12.763287 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:12.763494 (+ 207us) client_negotiation.cc:769] Sending connection context
0806 02:46:12.763630 (+ 136us) client_negotiation.cc:240] Negotiation successful
0806 02:46:12.764034 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":510,"threads_started":1}
I20260806 02:46:12.765290 9544 ts_manager.cc:194] Registered new tserver with Master: 7d735936b19c4d7aa4b8db79c7694f1d (127.1.210.65:33721)
I20260806 02:46:12.765483 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 9617
I20260806 02:46:12.765686 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance
I20260806 02:46:12.766496 1865 external_mini_cluster.cc:1428] Setting key 5c33d01508dda2abb6b2534b64dfe7ce
I20260806 02:46:12.767541 9544 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.210.65@KRBTEST.COM'} at 127.1.210.65:36521
WARNING: no policy specified for oryx/127.1.210.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.210.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.66.keytab.
Entry for principal oryx/127.1.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.66.keytab.
I20260806 02:46:12.823323 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.66:0
--local_ip_for_outbound_sockets=127.1.210.66
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:36511
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.66.keytab
--principal=oryx/127.1.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:37149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:46:13.171537 9759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:13.171838 9759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:13.171937 9759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:13.180052 9759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:13.180179 9759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:13.180266 9759 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:46:13.180351 9759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.66
I20260806 02:46:13.190400 9759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:37149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.66.keytab
--principal=oryx/127.1.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.210.126:36511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.66
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:46:13.191787 9759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:13.193153 9759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:13.207310 9764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:13.207293 9765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:13.207370 9767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:13.208380 9759 server_base.cc:1034] running on GCE node
I20260806 02:46:13.209115 9759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:13.210227 9759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:13.211531 9759 hybrid_clock.cc:648] HybridClock initialized: now 1785984373211443 us; error 54 us; skew 500 ppm
I20260806 02:46:13.211913 9759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:46:13 dist-test-slave-8tdl krb5kdc[9499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984373, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:13.215618 9759 init.cc:377] Logged in from keytab as oryx/127.1.210.66@KRBTEST.COM (short username oryx)
I20260806 02:46:13.217394 9759 webserver.cc:460] Webserver started at http://127.1.210.66:45409/ using document root <none> and password file <none>
I20260806 02:46:13.218334 9759 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:13.218482 9759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:13.218968 9759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:13.222054 9759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-1/data/instance:
uuid: "89153066de564aab9ff1c28037bb286c"
format_stamp: "Formatted at 2026-08-06 02:46:13 on dist-test-slave-8tdl"
server_key: "566b27f4e6d6252f0fa75463738f1595"
server_key_iv: "bfdd3dd0d358315d1ad70811db4db250"
server_key_version: "encryptionkey@0"
I20260806 02:46:13.222930 9759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance:
uuid: "89153066de564aab9ff1c28037bb286c"
format_stamp: "Formatted at 2026-08-06 02:46:13 on dist-test-slave-8tdl"
server_key: "566b27f4e6d6252f0fa75463738f1595"
server_key_iv: "bfdd3dd0d358315d1ad70811db4db250"
server_key_version: "encryptionkey@0"
I20260806 02:46:13.228302 9759 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260806 02:46:13.232353 9774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:13.234071 9759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:46:13.234261 9759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-1/wal
uuid: "89153066de564aab9ff1c28037bb286c"
format_stamp: "Formatted at 2026-08-06 02:46:13 on dist-test-slave-8tdl"
server_key: "566b27f4e6d6252f0fa75463738f1595"
server_key_iv: "bfdd3dd0d358315d1ad70811db4db250"
server_key_version: "encryptionkey@0"
I20260806 02:46:13.234517 9759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:13.303932 9759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:13.308779 9759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:13.309091 9759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:13.312232 9759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:13.312325 9759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:13.312522 9759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:13.312585 9759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:46:13 dist-test-slave-8tdl krb5kdc[9499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984373, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM
I20260806 02:46:13.343966 9783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:13.313101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:33147 (local address 127.1.210.126:36511)
0806 02:46:13.313791 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:13.313797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:13.315167 (+ 1370us) server_negotiation.cc:407] Connection header received
0806 02:46:13.317669 (+ 2502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:13.317675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:13.317780 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:13.317867 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:13.321368 (+ 3501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:13.322320 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:13.326098 (+ 3778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:13.327033 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:13.327197 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:13.334195 (+ 6998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:13.334206 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:13.334211 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:13.334283 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:13.338646 (+ 4363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:13.340075 (+ 1429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:13.340085 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:13.340091 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:13.340299 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:13.340996 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:13.341005 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:13.341011 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:13.341456 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:13.342849 (+ 1393us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:13.343028 (+ 179us) server_negotiation.cc:299] Negotiation successful
0806 02:46:13.343451 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":436,"threads_started":1}
I20260806 02:46:13.345738 9784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:13.313973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:36511 (local address 127.1.210.66:33147)
0806 02:46:13.314949 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:13.315022 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:13.317255 (+ 2233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:13.318543 (+ 1288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:13.318556 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:13.319841 (+ 1285us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:13.321050 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:13.321088 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:13.322539 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:13.322549 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:13.325850 (+ 3301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:13.325875 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:13.327605 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:13.327616 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:13.328019 (+ 403us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:13.329096 (+ 1077us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:13.329144 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:13.333721 (+ 4577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:13.338872 (+ 5151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:13.338886 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:13.338911 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:13.339846 (+ 935us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:13.340463 (+ 617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:13.340473 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:13.340476 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:13.340810 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:13.341648 (+ 838us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:13.341658 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:13.342368 (+ 710us) client_negotiation.cc:769] Sending connection context
0806 02:46:13.342818 (+ 450us) client_negotiation.cc:240] Negotiation successful
0806 02:46:13.343595 (+ 777us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":392,"threads_started":1}
I20260806 02:46:13.355868 9759 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.66:37491
I20260806 02:46:13.355895 9895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.66:37491 every 8 connection(s)
I20260806 02:46:13.357283 9759 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
I20260806 02:46:13.359975 9896 heartbeater.cc:347] Connected to a master server at 127.1.210.126:36511
I20260806 02:46:13.360244 9896 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:13.361076 9896 heartbeater.cc:511] Master 127.1.210.126:36511 requested a full tablet report, sending...
I20260806 02:46:13.361385 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 9759
I20260806 02:46:13.361565 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance
I20260806 02:46:13.362061 1865 external_mini_cluster.cc:1428] Setting key 7c410ddeccfc0f05258d7e4959a53fbf
I20260806 02:46:13.362915 9544 ts_manager.cc:194] Registered new tserver with Master: 89153066de564aab9ff1c28037bb286c (127.1.210.66:37491)
I20260806 02:46:13.364100 9544 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.210.66@KRBTEST.COM'} at 127.1.210.66:33147
I20260806 02:46:13.368041 9783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:13.353423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:34803 (local address 127.1.210.126:36511)
0806 02:46:13.353643 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:13.353652 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:13.354289 (+ 637us) server_negotiation.cc:407] Connection header received
0806 02:46:13.354525 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:13.354533 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:13.354630 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:13.354764 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:13.356007 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:13.356844 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:13.358433 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:13.359010 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:13.359160 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:13.361424 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:13.361433 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:13.361440 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:13.361488 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:13.365146 (+ 3658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:13.365908 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:13.365915 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:13.365919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:13.366046 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:13.366435 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:13.366441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:13.366444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:13.366851 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:13.367005 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:13.367461 (+ 456us) server_negotiation.cc:299] Negotiation successful
0806 02:46:13.367694 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 02:46:13.368161 9891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:13.353441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:36511 (local address 127.1.210.66:34803)
0806 02:46:13.354147 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:13.354167 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:13.354355 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:13.354950 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:13.354954 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:13.355480 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:13.355848 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:13.355857 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:13.356984 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:13.356990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:13.358207 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:13.358228 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:13.359235 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:13.359241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:13.359451 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:13.360132 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:13.360154 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:13.361159 (+ 1005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:13.365290 (+ 4131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:13.365295 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:13.365300 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:13.365773 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:13.366171 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:13.366176 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:13.366180 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:13.366319 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:13.366984 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:13.366989 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:13.367184 (+ 195us) client_negotiation.cc:769] Sending connection context
0806 02:46:13.367339 (+ 155us) client_negotiation.cc:240] Negotiation successful
0806 02:46:13.367678 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":347,"threads_started":1}
WARNING: no policy specified for oryx/127.1.210.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.210.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.67.keytab.
Entry for principal oryx/127.1.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.67.keytab.
I20260806 02:46:13.419394 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.67:0
--local_ip_for_outbound_sockets=127.1.210.67
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:36511
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.67.keytab
--principal=oryx/127.1.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:37149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:46:13.771404 9755 heartbeater.cc:503] Master 127.1.210.126:36511 was elected leader, sending a full tablet report...
W20260806 02:46:13.783636 9900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:13.784013 9900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:13.784116 9900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:13.791695 9900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:13.791824 9900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:13.791916 9900 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:46:13.792001 9900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.67
I20260806 02:46:13.802064 9900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:37149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.67.keytab
--principal=oryx/127.1.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.210.126:36511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.67
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:46:13.803442 9900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:13.804801 9900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:13.818409 9905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:13.818431 9906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:13.818655 9908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:13.819594 9900 server_base.cc:1034] running on GCE node
I20260806 02:46:13.820288 9900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:13.821210 9900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:13.822438 9900 hybrid_clock.cc:648] HybridClock initialized: now 1785984373822378 us; error 30 us; skew 500 ppm
I20260806 02:46:13.822824 9900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:46:13 dist-test-slave-8tdl krb5kdc[9499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984373, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:13.826210 9900 init.cc:377] Logged in from keytab as oryx/127.1.210.67@KRBTEST.COM (short username oryx)
I20260806 02:46:13.827730 9900 webserver.cc:460] Webserver started at http://127.1.210.67:33951/ using document root <none> and password file <none>
I20260806 02:46:13.828424 9900 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:13.828514 9900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:13.828830 9900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:13.832311 9900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-2/data/instance:
uuid: "1fa6b44ef46044d3a113266f1ebba9e4"
format_stamp: "Formatted at 2026-08-06 02:46:13 on dist-test-slave-8tdl"
server_key: "54a955bb8ff252ff2cb33267fd5f6299"
server_key_iv: "f0b188779ed98d14b5c8e64c9f3d073f"
server_key_version: "encryptionkey@0"
I20260806 02:46:13.833225 9900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance:
uuid: "1fa6b44ef46044d3a113266f1ebba9e4"
format_stamp: "Formatted at 2026-08-06 02:46:13 on dist-test-slave-8tdl"
server_key: "54a955bb8ff252ff2cb33267fd5f6299"
server_key_iv: "f0b188779ed98d14b5c8e64c9f3d073f"
server_key_version: "encryptionkey@0"
I20260806 02:46:13.839646 9900 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260806 02:46:13.843714 9915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:13.845352 9900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260806 02:46:13.845577 9900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-2/wal
uuid: "1fa6b44ef46044d3a113266f1ebba9e4"
format_stamp: "Formatted at 2026-08-06 02:46:13 on dist-test-slave-8tdl"
server_key: "54a955bb8ff252ff2cb33267fd5f6299"
server_key_iv: "f0b188779ed98d14b5c8e64c9f3d073f"
server_key_version: "encryptionkey@0"
I20260806 02:46:13.845861 9900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:13.876482 9900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:13.882764 9900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:13.883121 9900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:13.886206 9900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:13.886348 9900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:13.886551 9900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:13.886626 9900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:46:13 dist-test-slave-8tdl krb5kdc[9499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984373, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM
I20260806 02:46:13.921380 9924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:13.887146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:51553 (local address 127.1.210.126:36511)
0806 02:46:13.895150 (+ 8004us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:13.895155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:13.895175 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:46:13.895235 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:13.895239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:13.895336 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:13.895510 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:13.899080 (+ 3570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:13.900178 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:13.904388 (+ 4210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:13.905353 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:13.905505 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:13.912309 (+ 6804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:13.912328 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:13.912333 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:13.912386 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:13.916194 (+ 3808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:13.917628 (+ 1434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:13.917637 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:13.917644 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:13.917916 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:13.918580 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:13.918590 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:13.918596 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:13.919158 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:13.919399 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:13.920282 (+ 883us) server_negotiation.cc:299] Negotiation successful
0806 02:46:13.920663 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7851,"thread_start_us":7739,"threads_started":1}
I20260806 02:46:13.921947 9925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:13.888163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:36511 (local address 127.1.210.67:51553)
0806 02:46:13.889370 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:13.889435 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:13.893113 (+ 3678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:13.895998 (+ 2885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:13.896019 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:13.897339 (+ 1320us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:13.898624 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:13.898666 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:13.900448 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:13.900458 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:13.904093 (+ 3635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:13.904122 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:13.906097 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:13.906117 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:13.906586 (+ 469us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:13.907817 (+ 1231us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:13.907862 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:13.911773 (+ 3911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:13.916380 (+ 4607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:13.916391 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:13.916418 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:13.917423 (+ 1005us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:13.918069 (+ 646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:13.918077 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:13.918082 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:13.918424 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:13.919356 (+ 932us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:13.919362 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:13.919795 (+ 433us) client_negotiation.cc:769] Sending connection context
0806 02:46:13.920086 (+ 291us) client_negotiation.cc:240] Negotiation successful
0806 02:46:13.920686 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"thread_start_us":488,"threads_started":1}
I20260806 02:46:13.949091 9924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:13.929949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:55161 (local address 127.1.210.126:36511)
0806 02:46:13.930203 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:13.930212 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:13.931070 (+ 858us) server_negotiation.cc:407] Connection header received
0806 02:46:13.931370 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:13.931378 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:13.931481 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:13.931572 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:13.933549 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:13.934329 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:13.937060 (+ 2731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:13.937782 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:13.937976 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:13.941452 (+ 3476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:13.941459 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:13.941463 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:13.941504 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:13.945454 (+ 3950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:13.946625 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:13.946638 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:13.946641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:13.946856 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:13.947307 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:13.947312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:13.947314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:13.947775 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:13.947961 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:13.948420 (+ 459us) server_negotiation.cc:299] Negotiation successful
0806 02:46:13.948691 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 02:46:13.949205 10007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:13.929970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:36511 (local address 127.1.210.67:55161)
0806 02:46:13.930917 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:13.930947 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:13.931197 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:13.931688 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:13.931696 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:13.932727 (+ 1031us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:13.933326 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:13.933345 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:13.934488 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:13.934498 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:13.936815 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:13.936837 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:13.938098 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:13.938110 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:13.938409 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:13.939751 (+ 1342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:13.939775 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:13.941161 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:13.945722 (+ 4561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:13.945728 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:13.945733 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:13.946459 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:13.946997 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:13.947002 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:13.947006 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:13.947161 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:13.947900 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:13.947906 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:13.948115 (+ 209us) client_negotiation.cc:769] Sending connection context
0806 02:46:13.948260 (+ 145us) client_negotiation.cc:240] Negotiation successful
0806 02:46:13.948681 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":436,"threads_started":1}
I20260806 02:46:13.952407 9900 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.67:42521
I20260806 02:46:13.952435 10036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.67:42521 every 8 connection(s)
I20260806 02:46:13.953892 9900 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
I20260806 02:46:13.956312 10037 heartbeater.cc:347] Connected to a master server at 127.1.210.126:36511
I20260806 02:46:13.956640 10037 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:13.957368 10037 heartbeater.cc:511] Master 127.1.210.126:36511 requested a full tablet report, sending...
I20260806 02:46:13.958204 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 9900
I20260806 02:46:13.958411 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance
I20260806 02:46:13.959017 1865 external_mini_cluster.cc:1428] Setting key 7e837f91a5d878d50699184dd77548b3
I20260806 02:46:13.959263 9544 ts_manager.cc:194] Registered new tserver with Master: 1fa6b44ef46044d3a113266f1ebba9e4 (127.1.210.67:42521)
I20260806 02:46:13.960448 9544 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.210.67@KRBTEST.COM'} at 127.1.210.67:51553
I20260806 02:46:13.962428 1865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:46:13 dist-test-slave-8tdl krb5kdc[9499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984373, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 02:46:14 dist-test-slave-8tdl krb5kdc[9499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:14 dist-test-slave-8tdl krb5kdc[9499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:14.367856 9896 heartbeater.cc:503] Master 127.1.210.126:36511 was elected leader, sending a full tablet report...
I20260806 02:46:14.963474 10037 heartbeater.cc:503] Master 127.1.210.126:36511 was elected leader, sending a full tablet report...
W20260806 02:46:15.191152 10045 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.175s user 0.004s sys 0.014s
W20260806 02:46:15.192593 9924 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:14.009091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34346 (local address 127.1.210.126:36511)
0806 02:46:14.009278 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:14.009284 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:14.009509 (+ 225us) server_negotiation.cc:407] Connection header received
0806 02:46:14.009686 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:14.009690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:14.009760 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:14.009856 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:14.011694 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:14.012608 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:14.014095 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:14.014559 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:14.014689 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:15.192343 (+1177654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34346: BlockingRecv error: recv got EOF from 127.0.0.1:34346 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 06 02:46:15 dist-test-slave-8tdl krb5kdc[9499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984373, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM
I20260806 02:46:15.219461 9924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:15.197894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33726 (local address 127.1.210.126:36511)
0806 02:46:15.198163 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:15.198169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:15.198332 (+ 163us) server_negotiation.cc:407] Connection header received
0806 02:46:15.198592 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:15.198597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:15.198670 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:15.198840 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:15.200330 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.201091 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.202856 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.203402 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.203521 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:15.213062 (+ 9541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:15.213068 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:15.213071 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:15.213116 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:15.216092 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:15.217219 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:15.217224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:15.217226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:15.217370 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:15.217762 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:15.217766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:15.217768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:15.218128 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:15.218396 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:15.218814 (+ 418us) server_negotiation.cc:299] Negotiation successful
0806 02:46:15.219089 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260806 02:46:15.224514 9544 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:46:15.232882 9544 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:46:15.273890 10060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:15.261990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38874 (local address 127.1.210.65:33721)
0806 02:46:15.264313 (+ 2323us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:15.264326 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:15.265099 (+ 773us) server_negotiation.cc:407] Connection header received
0806 02:46:15.265205 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:15.265211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:15.265444 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:15.265649 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:46:15.266396 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.267471 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.270999 (+ 3528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.272213 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.272357 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:15.272579 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:15.273365 (+ 786us) server_negotiation.cc:299] Negotiation successful
0806 02:46:15.273550 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2096,"thread_start_us":2040,"threads_started":1}
I20260806 02:46:15.273952 10061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:15.261953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:33721 (local address 127.0.0.1:38874)
0806 02:46:15.264807 (+ 2854us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:15.264824 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:15.264957 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:15.265887 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:15.265891 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:15.265918 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:46:15.266216 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:15.266224 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.267621 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.267630 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:15.270819 (+ 3189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:15.270834 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.272804 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.272812 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:15.273082 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:15.273164 (+ 82us) client_negotiation.cc:769] Sending connection context
0806 02:46:15.273310 (+ 146us) client_negotiation.cc:240] Negotiation successful
0806 02:46:15.273563 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2732,"mutex_wait_us":248,"thread_start_us":2563,"threads_started":1}
I20260806 02:46:15.274390 10058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:15.246996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:37491 (local address 127.0.0.1:53318)
0806 02:46:15.263840 (+ 16844us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:15.263868 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:15.264048 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:15.264917 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:15.264925 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:15.264978 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:46:15.265619 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:15.265634 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.267675 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.267682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:15.270733 (+ 3051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:15.270749 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.272867 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.272874 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:15.273101 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:15.273170 (+ 69us) client_negotiation.cc:769] Sending connection context
0806 02:46:15.273286 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 02:46:15.273974 (+ 688us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16637,"mutex_wait_us":199,"spinlock_wait_cycles":4224,"thread_start_us":16437,"threads_started":1}
I20260806 02:46:15.274607 10056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:15.246641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:42521 (local address 127.0.0.1:55050)
0806 02:46:15.260940 (+ 14299us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:15.261017 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:15.261299 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:15.265129 (+ 3830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:15.265135 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:15.265179 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:46:15.265627 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:15.265634 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.267662 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.267667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:15.270343 (+ 2676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:15.270356 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.272751 (+ 2395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.272758 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:15.273032 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:15.273160 (+ 128us) client_negotiation.cc:769] Sending connection context
0806 02:46:15.273286 (+ 126us) client_negotiation.cc:240] Negotiation successful
0806 02:46:15.274167 (+ 881us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13906,"mutex_wait_us":178,"spinlock_wait_cycles":1280,"thread_start_us":13645,"threads_started":1}
I20260806 02:46:15.274813 10057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:15.246870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55050 (local address 127.1.210.67:42521)
0806 02:46:15.262152 (+ 15282us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:15.262164 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:15.262201 (+ 37us) server_negotiation.cc:407] Connection header received
0806 02:46:15.262344 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:15.262356 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:15.262754 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:15.263040 (+ 286us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:46:15.265832 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.267503 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.270527 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.272175 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.272332 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:15.272573 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:15.273443 (+ 870us) server_negotiation.cc:299] Negotiation successful
0806 02:46:15.273617 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15092,"thread_start_us":17249,"threads_started":1}
I20260806 02:46:15.275015 10059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:15.247333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53318 (local address 127.1.210.66:37491)
0806 02:46:15.259825 (+ 12492us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:15.259840 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:15.264005 (+ 4165us) server_negotiation.cc:407] Connection header received
0806 02:46:15.264398 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:15.264421 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:15.264667 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:15.264886 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:46:15.265830 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.267482 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.270908 (+ 3426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.272325 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.272510 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:15.272707 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:15.273444 (+ 737us) server_negotiation.cc:299] Negotiation successful
0806 02:46:15.273619 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12233,"thread_start_us":12518,"threads_started":1}
I20260806 02:46:15.278798 9683 tablet_service.cc:1467] Processing CreateTablet for tablet 8362e336961f4566bc31be47b1bed0b8 (DEFAULT_TABLE table=test-table [id=84fcf8fec22f465ba6988194a6bdbc83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:46:15.278800 9825 tablet_service.cc:1467] Processing CreateTablet for tablet 8362e336961f4566bc31be47b1bed0b8 (DEFAULT_TABLE table=test-table [id=84fcf8fec22f465ba6988194a6bdbc83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:46:15.280467 9683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8362e336961f4566bc31be47b1bed0b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:15.280467 9825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8362e336961f4566bc31be47b1bed0b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:15.285355 9967 tablet_service.cc:1467] Processing CreateTablet for tablet 8362e336961f4566bc31be47b1bed0b8 (DEFAULT_TABLE table=test-table [id=84fcf8fec22f465ba6988194a6bdbc83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:46:15.286942 9967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8362e336961f4566bc31be47b1bed0b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:15.297767 10062 tablet_bootstrap.cc:492] T 8362e336961f4566bc31be47b1bed0b8 P 89153066de564aab9ff1c28037bb286c: Bootstrap starting.
I20260806 02:46:15.298352 10063 tablet_bootstrap.cc:492] T 8362e336961f4566bc31be47b1bed0b8 P 7d735936b19c4d7aa4b8db79c7694f1d: Bootstrap starting.
I20260806 02:46:15.302534 10064 tablet_bootstrap.cc:492] T 8362e336961f4566bc31be47b1bed0b8 P 1fa6b44ef46044d3a113266f1ebba9e4: Bootstrap starting.
I20260806 02:46:15.302796 10062 tablet_bootstrap.cc:654] T 8362e336961f4566bc31be47b1bed0b8 P 89153066de564aab9ff1c28037bb286c: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:15.304337 10062 log.cc:824] T 8362e336961f4566bc31be47b1bed0b8 P 89153066de564aab9ff1c28037bb286c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:46:15.306551 10063 tablet_bootstrap.cc:654] T 8362e336961f4566bc31be47b1bed0b8 P 7d735936b19c4d7aa4b8db79c7694f1d: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:15.306551 10064 tablet_bootstrap.cc:654] T 8362e336961f4566bc31be47b1bed0b8 P 1fa6b44ef46044d3a113266f1ebba9e4: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:15.307945 10064 log.cc:824] T 8362e336961f4566bc31be47b1bed0b8 P 1fa6b44ef46044d3a113266f1ebba9e4: Log is configured to *not* fsync() on all Append() calls
I20260806 02:46:15.307955 10063 log.cc:824] T 8362e336961f4566bc31be47b1bed0b8 P 7d735936b19c4d7aa4b8db79c7694f1d: Log is configured to *not* fsync() on all Append() calls
I20260806 02:46:15.308487 10062 tablet_bootstrap.cc:492] T 8362e336961f4566bc31be47b1bed0b8 P 89153066de564aab9ff1c28037bb286c: No bootstrap required, opened a new log
I20260806 02:46:15.308851 10062 ts_tablet_manager.cc:1397] T 8362e336961f4566bc31be47b1bed0b8 P 89153066de564aab9ff1c28037bb286c: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260806 02:46:15.311506 10064 tablet_bootstrap.cc:492] T 8362e336961f4566bc31be47b1bed0b8 P 1fa6b44ef46044d3a113266f1ebba9e4: No bootstrap required, opened a new log
I20260806 02:46:15.311506 10063 tablet_bootstrap.cc:492] T 8362e336961f4566bc31be47b1bed0b8 P 7d735936b19c4d7aa4b8db79c7694f1d: No bootstrap required, opened a new log
I20260806 02:46:15.311781 10064 ts_tablet_manager.cc:1397] T 8362e336961f4566bc31be47b1bed0b8 P 1fa6b44ef46044d3a113266f1ebba9e4: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260806 02:46:15.311781 10063 ts_tablet_manager.cc:1397] T 8362e336961f4566bc31be47b1bed0b8 P 7d735936b19c4d7aa4b8db79c7694f1d: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.000s
I20260806 02:46:15.318086 10062 raft_consensus.cc:348] T 8362e336961f4566bc31be47b1bed0b8 P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } } peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } }
I20260806 02:46:15.318589 10062 raft_consensus.cc:374] T 8362e336961f4566bc31be47b1bed0b8 P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:15.318725 10062 raft_consensus.cc:729] T 8362e336961f4566bc31be47b1bed0b8 P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89153066de564aab9ff1c28037bb286c, State: Initialized, Role: FOLLOWER
I20260806 02:46:15.319481 10062 consensus_queue.cc:260] T 8362e336961f4566bc31be47b1bed0b8 P 89153066de564aab9ff1c28037bb286c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } } peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } }
I20260806 02:46:15.321483 10062 ts_tablet_manager.cc:1428] T 8362e336961f4566bc31be47b1bed0b8 P 89153066de564aab9ff1c28037bb286c: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260806 02:46:15.322968 10064 raft_consensus.cc:348] T 8362e336961f4566bc31be47b1bed0b8 P 1fa6b44ef46044d3a113266f1ebba9e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } } peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } }
I20260806 02:46:15.322966 10063 raft_consensus.cc:348] T 8362e336961f4566bc31be47b1bed0b8 P 7d735936b19c4d7aa4b8db79c7694f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } } peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } }
I20260806 02:46:15.323432 10063 raft_consensus.cc:374] T 8362e336961f4566bc31be47b1bed0b8 P 7d735936b19c4d7aa4b8db79c7694f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:15.323443 10064 raft_consensus.cc:374] T 8362e336961f4566bc31be47b1bed0b8 P 1fa6b44ef46044d3a113266f1ebba9e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:15.323549 10064 raft_consensus.cc:729] T 8362e336961f4566bc31be47b1bed0b8 P 1fa6b44ef46044d3a113266f1ebba9e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fa6b44ef46044d3a113266f1ebba9e4, State: Initialized, Role: FOLLOWER
I20260806 02:46:15.323549 10063 raft_consensus.cc:729] T 8362e336961f4566bc31be47b1bed0b8 P 7d735936b19c4d7aa4b8db79c7694f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d735936b19c4d7aa4b8db79c7694f1d, State: Initialized, Role: FOLLOWER
I20260806 02:46:15.324218 10064 consensus_queue.cc:260] T 8362e336961f4566bc31be47b1bed0b8 P 1fa6b44ef46044d3a113266f1ebba9e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } } peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } }
I20260806 02:46:15.324357 10063 consensus_queue.cc:260] T 8362e336961f4566bc31be47b1bed0b8 P 7d735936b19c4d7aa4b8db79c7694f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } } peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } }
I20260806 02:46:15.326560 10063 ts_tablet_manager.cc:1428] T 8362e336961f4566bc31be47b1bed0b8 P 7d735936b19c4d7aa4b8db79c7694f1d: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260806 02:46:15.328063 10064 ts_tablet_manager.cc:1428] T 8362e336961f4566bc31be47b1bed0b8 P 1fa6b44ef46044d3a113266f1ebba9e4: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260806 02:46:15.360152 10068 raft_consensus.cc:482] T 8362e336961f4566bc31be47b1bed0b8 P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 02:46:15.360751 9897 tablet.cc:2369] T 8362e336961f4566bc31be47b1bed0b8 P 89153066de564aab9ff1c28037bb286c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:46:15.360646 10068 raft_consensus.cc:504] T 8362e336961f4566bc31be47b1bed0b8 P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } } peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } }
I20260806 02:46:15.362920 10068 leader_election.cc:288] T 8362e336961f4566bc31be47b1bed0b8 P 89153066de564aab9ff1c28037bb286c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fa6b44ef46044d3a113266f1ebba9e4 (127.1.210.67:42521), 7d735936b19c4d7aa4b8db79c7694f1d (127.1.210.65:33721)
I20260806 02:46:15.370489 10057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:15.363265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:42811 (local address 127.1.210.67:42521)
0806 02:46:15.363484 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:15.363491 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:15.364315 (+ 824us) server_negotiation.cc:407] Connection header received
0806 02:46:15.364538 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:15.364544 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:15.364630 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:15.364774 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:46:15.365542 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.366832 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.368746 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.369528 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.369726 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:15.369802 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:15.370090 (+ 288us) server_negotiation.cc:299] Negotiation successful
0806 02:46:15.370220 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:46:15.370664 10071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:15.363303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:42521 (local address 127.1.210.66:42811)
0806 02:46:15.364175 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:15.364200 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:15.364358 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:15.364816 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:15.364821 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:15.364852 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:46:15.365354 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:15.365368 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.366978 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.366982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:15.368550 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:15.368565 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.369743 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.369749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:15.369913 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:15.369936 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 02:46:15.370043 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 02:46:15.370195 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":445,"threads_started":1}
I20260806 02:46:15.371284 10060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:15.364512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:54275 (local address 127.1.210.65:33721)
0806 02:46:15.364787 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:15.364795 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:15.365163 (+ 368us) server_negotiation.cc:407] Connection header received
0806 02:46:15.365371 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:15.365377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:15.365464 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:15.365561 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:46:15.366131 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.367207 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.369106 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.369958 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.370178 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:15.370258 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:15.370572 (+ 314us) server_negotiation.cc:299] Negotiation successful
0806 02:46:15.371019 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260806 02:46:15.371338 10072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:15.364329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:33721 (local address 127.1.210.66:54275)
0806 02:46:15.364986 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:15.365016 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:15.365204 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:15.365586 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:15.365591 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:15.365615 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:46:15.365950 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:15.365961 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.367349 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.367356 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:15.368915 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:15.368928 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.370208 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.370215 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:15.370389 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:15.370418 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 02:46:15.370542 (+ 124us) client_negotiation.cc:240] Negotiation successful
0806 02:46:15.370681 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":458,"threads_started":1}
I20260806 02:46:15.371860 9987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8362e336961f4566bc31be47b1bed0b8" candidate_uuid: "89153066de564aab9ff1c28037bb286c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" is_pre_election: true
I20260806 02:46:15.372308 9704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8362e336961f4566bc31be47b1bed0b8" candidate_uuid: "89153066de564aab9ff1c28037bb286c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" is_pre_election: true
I20260806 02:46:15.372491 9987 raft_consensus.cc:2454] T 8362e336961f4566bc31be47b1bed0b8 P 1fa6b44ef46044d3a113266f1ebba9e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89153066de564aab9ff1c28037bb286c in term 0.
I20260806 02:46:15.372879 9704 raft_consensus.cc:2454] T 8362e336961f4566bc31be47b1bed0b8 P 7d735936b19c4d7aa4b8db79c7694f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89153066de564aab9ff1c28037bb286c in term 0.
I20260806 02:46:15.373306 9775 leader_election.cc:302] T 8362e336961f4566bc31be47b1bed0b8 P 89153066de564aab9ff1c28037bb286c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fa6b44ef46044d3a113266f1ebba9e4, 89153066de564aab9ff1c28037bb286c; no voters:
I20260806 02:46:15.373726 10068 raft_consensus.cc:2781] T 8362e336961f4566bc31be47b1bed0b8 P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:46:15.373893 10068 raft_consensus.cc:482] T 8362e336961f4566bc31be47b1bed0b8 P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:46:15.374030 10068 raft_consensus.cc:3037] T 8362e336961f4566bc31be47b1bed0b8 P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:15.376533 10068 raft_consensus.cc:504] T 8362e336961f4566bc31be47b1bed0b8 P 89153066de564aab9ff1c28037bb286c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } } peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } }
I20260806 02:46:15.377164 10068 leader_election.cc:288] T 8362e336961f4566bc31be47b1bed0b8 P 89153066de564aab9ff1c28037bb286c [CANDIDATE]: Term 1 election: Requested vote from peers 1fa6b44ef46044d3a113266f1ebba9e4 (127.1.210.67:42521), 7d735936b19c4d7aa4b8db79c7694f1d (127.1.210.65:33721)
I20260806 02:46:15.377905 9987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8362e336961f4566bc31be47b1bed0b8" candidate_uuid: "89153066de564aab9ff1c28037bb286c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fa6b44ef46044d3a113266f1ebba9e4"
I20260806 02:46:15.377905 9704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8362e336961f4566bc31be47b1bed0b8" candidate_uuid: "89153066de564aab9ff1c28037bb286c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d735936b19c4d7aa4b8db79c7694f1d"
I20260806 02:46:15.378165 9704 raft_consensus.cc:3037] T 8362e336961f4566bc31be47b1bed0b8 P 7d735936b19c4d7aa4b8db79c7694f1d [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:15.378165 9987 raft_consensus.cc:3037] T 8362e336961f4566bc31be47b1bed0b8 P 1fa6b44ef46044d3a113266f1ebba9e4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:15.381618 9704 raft_consensus.cc:2454] T 8362e336961f4566bc31be47b1bed0b8 P 7d735936b19c4d7aa4b8db79c7694f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89153066de564aab9ff1c28037bb286c in term 1.
I20260806 02:46:15.381615 9987 raft_consensus.cc:2454] T 8362e336961f4566bc31be47b1bed0b8 P 1fa6b44ef46044d3a113266f1ebba9e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89153066de564aab9ff1c28037bb286c in term 1.
I20260806 02:46:15.382282 9775 leader_election.cc:302] T 8362e336961f4566bc31be47b1bed0b8 P 89153066de564aab9ff1c28037bb286c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d735936b19c4d7aa4b8db79c7694f1d, 89153066de564aab9ff1c28037bb286c; no voters:
I20260806 02:46:15.382616 10068 raft_consensus.cc:2781] T 8362e336961f4566bc31be47b1bed0b8 P 89153066de564aab9ff1c28037bb286c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:46:15.383176 10068 raft_consensus.cc:686] T 8362e336961f4566bc31be47b1bed0b8 P 89153066de564aab9ff1c28037bb286c [term 1 LEADER]: Becoming Leader. State: Replica: 89153066de564aab9ff1c28037bb286c, State: Running, Role: LEADER
I20260806 02:46:15.383733 10068 consensus_queue.cc:237] T 8362e336961f4566bc31be47b1bed0b8 P 89153066de564aab9ff1c28037bb286c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } } peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } }
I20260806 02:46:15.388948 9542 catalog_manager.cc:5495] T 8362e336961f4566bc31be47b1bed0b8 P 89153066de564aab9ff1c28037bb286c reported cstate change: term changed from 0 to 1, leader changed from <none> to 89153066de564aab9ff1c28037bb286c (127.1.210.66). New cstate: current_term: 1 leader_uuid: "89153066de564aab9ff1c28037bb286c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } health_report { overall_health: HEALTHY } } }
I20260806 02:46:15.435683 10059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:15.429516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53332 (local address 127.1.210.66:37491)
0806 02:46:15.429770 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:15.429776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:15.429894 (+ 118us) server_negotiation.cc:407] Connection header received
0806 02:46:15.430174 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:15.430178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:15.430259 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:15.430425 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:46:15.430908 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.432100 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.433496 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.433888 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.434030 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:15.434491 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:46:15.435092 (+ 601us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:46:15.435264 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:15.435321 (+ 57us) server_negotiation.cc:299] Negotiation successful
0806 02:46:15.435403 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260806 02:46:15.444862 9704 raft_consensus.cc:1261] T 8362e336961f4566bc31be47b1bed0b8 P 7d735936b19c4d7aa4b8db79c7694f1d [term 1 FOLLOWER]: Refusing update from remote peer 89153066de564aab9ff1c28037bb286c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:46:15.444859 9987 raft_consensus.cc:1261] T 8362e336961f4566bc31be47b1bed0b8 P 1fa6b44ef46044d3a113266f1ebba9e4 [term 1 FOLLOWER]: Refusing update from remote peer 89153066de564aab9ff1c28037bb286c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:46:15.446041 10068 consensus_queue.cc:1035] T 8362e336961f4566bc31be47b1bed0b8 P 89153066de564aab9ff1c28037bb286c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:46:15.446385 10073 consensus_queue.cc:1035] T 8362e336961f4566bc31be47b1bed0b8 P 89153066de564aab9ff1c28037bb286c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 02:46:15.457541 10038 tablet.cc:2369] T 8362e336961f4566bc31be47b1bed0b8 P 1fa6b44ef46044d3a113266f1ebba9e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:46:15.461866 10077 mvcc.cc:205] Tried to move back new op lower bound from 7315392001810321408 to 7315392001575608320. Current Snapshot: MvccSnapshot[applied={T|T < 7315392001810321408}]
I20260806 02:46:15.477699 9542 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:46:15.478102 9542 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:46:15.482590 9542 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e49507f75394515a4d7645a0f7848c1: Sending DeleteTablet for 3 replicas of tablet 8362e336961f4566bc31be47b1bed0b8
I20260806 02:46:15.484146 9967 tablet_service.cc:1514] Processing DeleteTablet for tablet 8362e336961f4566bc31be47b1bed0b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:46:15 UTC) from {username='oryx', principal='oryx/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:55050
I20260806 02:46:15.484153 9825 tablet_service.cc:1514] Processing DeleteTablet for tablet 8362e336961f4566bc31be47b1bed0b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:46:15 UTC) from {username='oryx', principal='oryx/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:53318
I20260806 02:46:15.485287 9683 tablet_service.cc:1514] Processing DeleteTablet for tablet 8362e336961f4566bc31be47b1bed0b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:46:15 UTC) from {username='oryx', principal='oryx/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:38874
I20260806 02:46:15.485466 9543 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:46:15.486719 9543 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:46:15.498096 9824 tablet_service.cc:1467] Processing CreateTablet for tablet b6ba3916e6f94e8184289fc2323f520d (DEFAULT_TABLE table=test-table [id=3e2ab77cf7454a60958abf6b6a9a718c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:46:15.499074 9824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6ba3916e6f94e8184289fc2323f520d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:15.504155 9966 tablet_service.cc:1467] Processing CreateTablet for tablet b6ba3916e6f94e8184289fc2323f520d (DEFAULT_TABLE table=test-table [id=3e2ab77cf7454a60958abf6b6a9a718c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:46:15.504972 9966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6ba3916e6f94e8184289fc2323f520d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:15.507772 9682 tablet_service.cc:1467] Processing CreateTablet for tablet b6ba3916e6f94e8184289fc2323f520d (DEFAULT_TABLE table=test-table [id=3e2ab77cf7454a60958abf6b6a9a718c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:46:15.508540 9682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6ba3916e6f94e8184289fc2323f520d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:15.518324 10086 tablet_replica.cc:330] T 8362e336961f4566bc31be47b1bed0b8 P 1fa6b44ef46044d3a113266f1ebba9e4: stopping tablet replica
I20260806 02:46:15.519104 10086 raft_consensus.cc:2229] T 8362e336961f4566bc31be47b1bed0b8 P 1fa6b44ef46044d3a113266f1ebba9e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:46:15.519618 10063 tablet_bootstrap.cc:492] T b6ba3916e6f94e8184289fc2323f520d P 7d735936b19c4d7aa4b8db79c7694f1d: Bootstrap starting.
I20260806 02:46:15.519708 10086 raft_consensus.cc:2258] T 8362e336961f4566bc31be47b1bed0b8 P 1fa6b44ef46044d3a113266f1ebba9e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:46:15.521373 10063 tablet_bootstrap.cc:654] T b6ba3916e6f94e8184289fc2323f520d P 7d735936b19c4d7aa4b8db79c7694f1d: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:15.524540 10086 ts_tablet_manager.cc:1905] T 8362e336961f4566bc31be47b1bed0b8 P 1fa6b44ef46044d3a113266f1ebba9e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:46:15.534808 10063 tablet_bootstrap.cc:492] T b6ba3916e6f94e8184289fc2323f520d P 7d735936b19c4d7aa4b8db79c7694f1d: No bootstrap required, opened a new log
I20260806 02:46:15.535214 10063 ts_tablet_manager.cc:1397] T b6ba3916e6f94e8184289fc2323f520d P 7d735936b19c4d7aa4b8db79c7694f1d: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.000s
I20260806 02:46:15.536293 10063 raft_consensus.cc:348] T b6ba3916e6f94e8184289fc2323f520d P 7d735936b19c4d7aa4b8db79c7694f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } }
I20260806 02:46:15.536638 10063 raft_consensus.cc:374] T b6ba3916e6f94e8184289fc2323f520d P 7d735936b19c4d7aa4b8db79c7694f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:15.536791 10063 raft_consensus.cc:729] T b6ba3916e6f94e8184289fc2323f520d P 7d735936b19c4d7aa4b8db79c7694f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d735936b19c4d7aa4b8db79c7694f1d, State: Initialized, Role: FOLLOWER
I20260806 02:46:15.537145 10063 consensus_queue.cc:260] T b6ba3916e6f94e8184289fc2323f520d P 7d735936b19c4d7aa4b8db79c7694f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } }
I20260806 02:46:15.538008 10062 tablet_bootstrap.cc:492] T b6ba3916e6f94e8184289fc2323f520d P 89153066de564aab9ff1c28037bb286c: Bootstrap starting.
I20260806 02:46:15.538831 10087 tablet_replica.cc:330] T 8362e336961f4566bc31be47b1bed0b8 P 7d735936b19c4d7aa4b8db79c7694f1d: stopping tablet replica
I20260806 02:46:15.539489 10087 raft_consensus.cc:2229] T 8362e336961f4566bc31be47b1bed0b8 P 7d735936b19c4d7aa4b8db79c7694f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:46:15.540000 10062 tablet_bootstrap.cc:654] T b6ba3916e6f94e8184289fc2323f520d P 89153066de564aab9ff1c28037bb286c: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:15.540035 10087 raft_consensus.cc:2258] T 8362e336961f4566bc31be47b1bed0b8 P 7d735936b19c4d7aa4b8db79c7694f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:46:15.540917 10063 ts_tablet_manager.cc:1428] T b6ba3916e6f94e8184289fc2323f520d P 7d735936b19c4d7aa4b8db79c7694f1d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260806 02:46:15.541767 10085 tablet_replica.cc:330] T 8362e336961f4566bc31be47b1bed0b8 P 89153066de564aab9ff1c28037bb286c: stopping tablet replica
I20260806 02:46:15.542251 10085 raft_consensus.cc:2229] T 8362e336961f4566bc31be47b1bed0b8 P 89153066de564aab9ff1c28037bb286c [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:46:15.543013 10085 raft_consensus.cc:2258] T 8362e336961f4566bc31be47b1bed0b8 P 89153066de564aab9ff1c28037bb286c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:46:15.543735 10087 ts_tablet_manager.cc:1905] T 8362e336961f4566bc31be47b1bed0b8 P 7d735936b19c4d7aa4b8db79c7694f1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:46:15.543939 10064 tablet_bootstrap.cc:492] T b6ba3916e6f94e8184289fc2323f520d P 1fa6b44ef46044d3a113266f1ebba9e4: Bootstrap starting.
I20260806 02:46:15.544310 10086 ts_tablet_manager.cc:1918] T 8362e336961f4566bc31be47b1bed0b8 P 1fa6b44ef46044d3a113266f1ebba9e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:46:15.544554 10086 log.cc:1196] T 8362e336961f4566bc31be47b1bed0b8 P 1fa6b44ef46044d3a113266f1ebba9e4: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-2/wal/wals/8362e336961f4566bc31be47b1bed0b8
I20260806 02:46:15.545223 10086 ts_tablet_manager.cc:1939] T 8362e336961f4566bc31be47b1bed0b8 P 1fa6b44ef46044d3a113266f1ebba9e4: Deleting consensus metadata
I20260806 02:46:15.545214 10085 ts_tablet_manager.cc:1905] T 8362e336961f4566bc31be47b1bed0b8 P 89153066de564aab9ff1c28037bb286c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:46:15.545776 10064 tablet_bootstrap.cc:654] T b6ba3916e6f94e8184289fc2323f520d P 1fa6b44ef46044d3a113266f1ebba9e4: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:15.549448 9528 catalog_manager.cc:4841] TS 1fa6b44ef46044d3a113266f1ebba9e4 (127.1.210.67:42521): tablet 8362e336961f4566bc31be47b1bed0b8 (table test-table [id=84fcf8fec22f465ba6988194a6bdbc83]) successfully deleted
I20260806 02:46:15.557485 10087 ts_tablet_manager.cc:1918] T 8362e336961f4566bc31be47b1bed0b8 P 7d735936b19c4d7aa4b8db79c7694f1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:46:15.557737 10087 log.cc:1196] T 8362e336961f4566bc31be47b1bed0b8 P 7d735936b19c4d7aa4b8db79c7694f1d: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-0/wal/wals/8362e336961f4566bc31be47b1bed0b8
I20260806 02:46:15.558127 10085 ts_tablet_manager.cc:1918] T 8362e336961f4566bc31be47b1bed0b8 P 89153066de564aab9ff1c28037bb286c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:46:15.558365 10087 ts_tablet_manager.cc:1939] T 8362e336961f4566bc31be47b1bed0b8 P 7d735936b19c4d7aa4b8db79c7694f1d: Deleting consensus metadata
I20260806 02:46:15.558430 10062 tablet_bootstrap.cc:492] T b6ba3916e6f94e8184289fc2323f520d P 89153066de564aab9ff1c28037bb286c: No bootstrap required, opened a new log
I20260806 02:46:15.558482 10085 log.cc:1196] T 8362e336961f4566bc31be47b1bed0b8 P 89153066de564aab9ff1c28037bb286c: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-1/wal/wals/8362e336961f4566bc31be47b1bed0b8
I20260806 02:46:15.558625 10062 ts_tablet_manager.cc:1397] T b6ba3916e6f94e8184289fc2323f520d P 89153066de564aab9ff1c28037bb286c: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.005s
I20260806 02:46:15.559294 10085 ts_tablet_manager.cc:1939] T 8362e336961f4566bc31be47b1bed0b8 P 89153066de564aab9ff1c28037bb286c: Deleting consensus metadata
I20260806 02:46:15.560367 10064 tablet_bootstrap.cc:492] T b6ba3916e6f94e8184289fc2323f520d P 1fa6b44ef46044d3a113266f1ebba9e4: No bootstrap required, opened a new log
I20260806 02:46:15.560567 10064 ts_tablet_manager.cc:1397] T b6ba3916e6f94e8184289fc2323f520d P 1fa6b44ef46044d3a113266f1ebba9e4: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.011s
I20260806 02:46:15.560524 9528 catalog_manager.cc:4841] TS 7d735936b19c4d7aa4b8db79c7694f1d (127.1.210.65:33721): tablet 8362e336961f4566bc31be47b1bed0b8 (table test-table [id=84fcf8fec22f465ba6988194a6bdbc83]) successfully deleted
I20260806 02:46:15.560660 10062 raft_consensus.cc:348] T b6ba3916e6f94e8184289fc2323f520d P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } }
I20260806 02:46:15.560969 10062 raft_consensus.cc:374] T b6ba3916e6f94e8184289fc2323f520d P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:15.561190 10062 raft_consensus.cc:729] T b6ba3916e6f94e8184289fc2323f520d P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89153066de564aab9ff1c28037bb286c, State: Initialized, Role: FOLLOWER
I20260806 02:46:15.561484 10064 raft_consensus.cc:348] T b6ba3916e6f94e8184289fc2323f520d P 1fa6b44ef46044d3a113266f1ebba9e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } }
I20260806 02:46:15.561558 10062 consensus_queue.cc:260] T b6ba3916e6f94e8184289fc2323f520d P 89153066de564aab9ff1c28037bb286c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } }
I20260806 02:46:15.561729 10064 raft_consensus.cc:374] T b6ba3916e6f94e8184289fc2323f520d P 1fa6b44ef46044d3a113266f1ebba9e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:15.561829 10064 raft_consensus.cc:729] T b6ba3916e6f94e8184289fc2323f520d P 1fa6b44ef46044d3a113266f1ebba9e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fa6b44ef46044d3a113266f1ebba9e4, State: Initialized, Role: FOLLOWER
I20260806 02:46:15.562184 10062 ts_tablet_manager.cc:1428] T b6ba3916e6f94e8184289fc2323f520d P 89153066de564aab9ff1c28037bb286c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260806 02:46:15.562103 10064 consensus_queue.cc:260] T b6ba3916e6f94e8184289fc2323f520d P 1fa6b44ef46044d3a113266f1ebba9e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } }
I20260806 02:46:15.563061 9530 catalog_manager.cc:4841] TS 89153066de564aab9ff1c28037bb286c (127.1.210.66:37491): tablet 8362e336961f4566bc31be47b1bed0b8 (table test-table [id=84fcf8fec22f465ba6988194a6bdbc83]) successfully deleted
I20260806 02:46:15.563364 10064 ts_tablet_manager.cc:1428] T b6ba3916e6f94e8184289fc2323f520d P 1fa6b44ef46044d3a113266f1ebba9e4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260806 02:46:15.678630 10068 raft_consensus.cc:482] T b6ba3916e6f94e8184289fc2323f520d P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:46:15.678967 10068 raft_consensus.cc:504] T b6ba3916e6f94e8184289fc2323f520d P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } }
I20260806 02:46:15.679701 10068 leader_election.cc:288] T b6ba3916e6f94e8184289fc2323f520d P 89153066de564aab9ff1c28037bb286c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d735936b19c4d7aa4b8db79c7694f1d (127.1.210.65:33721), 1fa6b44ef46044d3a113266f1ebba9e4 (127.1.210.67:42521)
I20260806 02:46:15.680418 9987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ba3916e6f94e8184289fc2323f520d" candidate_uuid: "89153066de564aab9ff1c28037bb286c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" is_pre_election: true
I20260806 02:46:15.680459 9704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ba3916e6f94e8184289fc2323f520d" candidate_uuid: "89153066de564aab9ff1c28037bb286c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" is_pre_election: true
I20260806 02:46:15.680694 9987 raft_consensus.cc:2454] T b6ba3916e6f94e8184289fc2323f520d P 1fa6b44ef46044d3a113266f1ebba9e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89153066de564aab9ff1c28037bb286c in term 0.
I20260806 02:46:15.680749 9704 raft_consensus.cc:2454] T b6ba3916e6f94e8184289fc2323f520d P 7d735936b19c4d7aa4b8db79c7694f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89153066de564aab9ff1c28037bb286c in term 0.
I20260806 02:46:15.681267 9775 leader_election.cc:302] T b6ba3916e6f94e8184289fc2323f520d P 89153066de564aab9ff1c28037bb286c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d735936b19c4d7aa4b8db79c7694f1d, 89153066de564aab9ff1c28037bb286c; no voters:
I20260806 02:46:15.681563 10068 raft_consensus.cc:2781] T b6ba3916e6f94e8184289fc2323f520d P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:46:15.681695 10068 raft_consensus.cc:482] T b6ba3916e6f94e8184289fc2323f520d P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:46:15.681802 10068 raft_consensus.cc:3037] T b6ba3916e6f94e8184289fc2323f520d P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:15.684281 10068 raft_consensus.cc:504] T b6ba3916e6f94e8184289fc2323f520d P 89153066de564aab9ff1c28037bb286c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } }
I20260806 02:46:15.684978 10068 leader_election.cc:288] T b6ba3916e6f94e8184289fc2323f520d P 89153066de564aab9ff1c28037bb286c [CANDIDATE]: Term 1 election: Requested vote from peers 7d735936b19c4d7aa4b8db79c7694f1d (127.1.210.65:33721), 1fa6b44ef46044d3a113266f1ebba9e4 (127.1.210.67:42521)
I20260806 02:46:15.685556 9704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ba3916e6f94e8184289fc2323f520d" candidate_uuid: "89153066de564aab9ff1c28037bb286c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d735936b19c4d7aa4b8db79c7694f1d"
I20260806 02:46:15.685670 9987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ba3916e6f94e8184289fc2323f520d" candidate_uuid: "89153066de564aab9ff1c28037bb286c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fa6b44ef46044d3a113266f1ebba9e4"
I20260806 02:46:15.685814 9704 raft_consensus.cc:3037] T b6ba3916e6f94e8184289fc2323f520d P 7d735936b19c4d7aa4b8db79c7694f1d [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:15.685910 9987 raft_consensus.cc:3037] T b6ba3916e6f94e8184289fc2323f520d P 1fa6b44ef46044d3a113266f1ebba9e4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:15.688218 9704 raft_consensus.cc:2454] T b6ba3916e6f94e8184289fc2323f520d P 7d735936b19c4d7aa4b8db79c7694f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89153066de564aab9ff1c28037bb286c in term 1.
I20260806 02:46:15.688710 9775 leader_election.cc:302] T b6ba3916e6f94e8184289fc2323f520d P 89153066de564aab9ff1c28037bb286c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d735936b19c4d7aa4b8db79c7694f1d, 89153066de564aab9ff1c28037bb286c; no voters:
I20260806 02:46:15.689054 10068 raft_consensus.cc:2781] T b6ba3916e6f94e8184289fc2323f520d P 89153066de564aab9ff1c28037bb286c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:46:15.689059 9987 raft_consensus.cc:2454] T b6ba3916e6f94e8184289fc2323f520d P 1fa6b44ef46044d3a113266f1ebba9e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89153066de564aab9ff1c28037bb286c in term 1.
I20260806 02:46:15.689251 10068 raft_consensus.cc:686] T b6ba3916e6f94e8184289fc2323f520d P 89153066de564aab9ff1c28037bb286c [term 1 LEADER]: Becoming Leader. State: Replica: 89153066de564aab9ff1c28037bb286c, State: Running, Role: LEADER
I20260806 02:46:15.689723 10068 consensus_queue.cc:237] T b6ba3916e6f94e8184289fc2323f520d P 89153066de564aab9ff1c28037bb286c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } }
I20260806 02:46:15.692615 9542 catalog_manager.cc:5495] T b6ba3916e6f94e8184289fc2323f520d P 89153066de564aab9ff1c28037bb286c reported cstate change: term changed from 0 to 1, leader changed from <none> to 89153066de564aab9ff1c28037bb286c (127.1.210.66). New cstate: current_term: 1 leader_uuid: "89153066de564aab9ff1c28037bb286c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } health_report { overall_health: UNKNOWN } } }
Aug 06 02:46:15 dist-test-slave-8tdl krb5kdc[9499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984371, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.126@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM
W20260806 02:46:15.765172 9756 tablet.cc:2369] T b6ba3916e6f94e8184289fc2323f520d P 7d735936b19c4d7aa4b8db79c7694f1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:46:15.768982 10098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:15.753462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33730 (local address 127.1.210.126:36511)
0806 02:46:15.754615 (+ 1153us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:15.754619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:15.754636 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:46:15.754725 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:15.754732 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:15.754802 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:15.754945 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:15.756284 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.757244 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.758517 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.759016 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.759165 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:15.762963 (+ 3798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:15.762969 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:15.762972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:15.763005 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:15.765858 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:15.766593 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:15.766601 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:15.766607 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:15.766825 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:15.767211 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:15.767219 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:15.767222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:15.767521 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:15.767714 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:15.768194 (+ 480us) server_negotiation.cc:299] Negotiation successful
0806 02:46:15.768492 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":1024,"thread_start_us":783,"threads_started":1}
I20260806 02:46:15.769022 10097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:15.753449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:36511 (local address 127.0.0.1:33730)
0806 02:46:15.754198 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:15.754229 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:15.754428 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:15.754968 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:15.754972 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:15.755838 (+ 866us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:15.756119 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:15.756129 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.757368 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.757376 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:15.758352 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:15.758364 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.759211 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.759217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:15.759422 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:15.760210 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:15.760228 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:15.762721 (+ 2493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:15.766009 (+ 3288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:15.766028 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:15.766036 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:15.766469 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:15.766941 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:15.766946 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:15.766950 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:15.767108 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:15.767670 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:15.767678 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:15.767908 (+ 230us) client_negotiation.cc:769] Sending connection context
0806 02:46:15.768116 (+ 208us) client_negotiation.cc:240] Negotiation successful
0806 02:46:15.768619 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":201,"thread_start_us":390,"threads_started":1}
I20260806 02:46:15.775897 9542 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:33730:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 02:46:15.777133 9542 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:46:15.785435 9966 tablet_service.cc:1467] Processing CreateTablet for tablet f1d165b802d34f48ab32c47bfdd03cca (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1a7c6661ec094a10beb94bbd456bc5e8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:46:15.785416 9825 tablet_service.cc:1467] Processing CreateTablet for tablet f1d165b802d34f48ab32c47bfdd03cca (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1a7c6661ec094a10beb94bbd456bc5e8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:46:15.785396 9683 tablet_service.cc:1467] Processing CreateTablet for tablet f1d165b802d34f48ab32c47bfdd03cca (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1a7c6661ec094a10beb94bbd456bc5e8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:46:15.786087 9966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1d165b802d34f48ab32c47bfdd03cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:15.786111 9825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1d165b802d34f48ab32c47bfdd03cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:15.786106 9683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1d165b802d34f48ab32c47bfdd03cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:15.793912 10064 tablet_bootstrap.cc:492] T f1d165b802d34f48ab32c47bfdd03cca P 1fa6b44ef46044d3a113266f1ebba9e4: Bootstrap starting.
I20260806 02:46:15.795482 10062 tablet_bootstrap.cc:492] T f1d165b802d34f48ab32c47bfdd03cca P 89153066de564aab9ff1c28037bb286c: Bootstrap starting.
I20260806 02:46:15.795907 10064 tablet_bootstrap.cc:654] T f1d165b802d34f48ab32c47bfdd03cca P 1fa6b44ef46044d3a113266f1ebba9e4: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:15.797500 10062 tablet_bootstrap.cc:654] T f1d165b802d34f48ab32c47bfdd03cca P 89153066de564aab9ff1c28037bb286c: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:15.799844 10063 tablet_bootstrap.cc:492] T f1d165b802d34f48ab32c47bfdd03cca P 7d735936b19c4d7aa4b8db79c7694f1d: Bootstrap starting.
I20260806 02:46:15.801570 10063 tablet_bootstrap.cc:654] T f1d165b802d34f48ab32c47bfdd03cca P 7d735936b19c4d7aa4b8db79c7694f1d: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:15.805416 10064 tablet_bootstrap.cc:492] T f1d165b802d34f48ab32c47bfdd03cca P 1fa6b44ef46044d3a113266f1ebba9e4: No bootstrap required, opened a new log
I20260806 02:46:15.805611 10064 ts_tablet_manager.cc:1397] T f1d165b802d34f48ab32c47bfdd03cca P 1fa6b44ef46044d3a113266f1ebba9e4: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20260806 02:46:15.806401 10064 raft_consensus.cc:348] T f1d165b802d34f48ab32c47bfdd03cca P 1fa6b44ef46044d3a113266f1ebba9e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } }
I20260806 02:46:15.806695 10064 raft_consensus.cc:374] T f1d165b802d34f48ab32c47bfdd03cca P 1fa6b44ef46044d3a113266f1ebba9e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:15.806774 10064 raft_consensus.cc:729] T f1d165b802d34f48ab32c47bfdd03cca P 1fa6b44ef46044d3a113266f1ebba9e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fa6b44ef46044d3a113266f1ebba9e4, State: Initialized, Role: FOLLOWER
I20260806 02:46:15.807035 10064 consensus_queue.cc:260] T f1d165b802d34f48ab32c47bfdd03cca P 1fa6b44ef46044d3a113266f1ebba9e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } }
I20260806 02:46:15.807399 10062 tablet_bootstrap.cc:492] T f1d165b802d34f48ab32c47bfdd03cca P 89153066de564aab9ff1c28037bb286c: No bootstrap required, opened a new log
I20260806 02:46:15.807734 10062 ts_tablet_manager.cc:1397] T f1d165b802d34f48ab32c47bfdd03cca P 89153066de564aab9ff1c28037bb286c: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260806 02:46:15.807497 10069 tablet_replica.cc:439] T f1d165b802d34f48ab32c47bfdd03cca P 1fa6b44ef46044d3a113266f1ebba9e4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } } }
I20260806 02:46:15.807878 10069 tablet_replica.cc:442] T f1d165b802d34f48ab32c47bfdd03cca P 1fa6b44ef46044d3a113266f1ebba9e4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:46:15.809060 10062 raft_consensus.cc:348] T f1d165b802d34f48ab32c47bfdd03cca P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } }
I20260806 02:46:15.809544 10062 raft_consensus.cc:374] T f1d165b802d34f48ab32c47bfdd03cca P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:15.809720 10062 raft_consensus.cc:729] T f1d165b802d34f48ab32c47bfdd03cca P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89153066de564aab9ff1c28037bb286c, State: Initialized, Role: FOLLOWER
I20260806 02:46:15.810207 10062 consensus_queue.cc:260] T f1d165b802d34f48ab32c47bfdd03cca P 89153066de564aab9ff1c28037bb286c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } }
I20260806 02:46:15.810987 10068 tablet_replica.cc:439] T f1d165b802d34f48ab32c47bfdd03cca P 89153066de564aab9ff1c28037bb286c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } } }
I20260806 02:46:15.811512 10068 tablet_replica.cc:442] T f1d165b802d34f48ab32c47bfdd03cca P 89153066de564aab9ff1c28037bb286c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:46:15.812757 10062 ts_tablet_manager.cc:1428] T f1d165b802d34f48ab32c47bfdd03cca P 89153066de564aab9ff1c28037bb286c: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260806 02:46:15.814832 10064 ts_tablet_manager.cc:1428] T f1d165b802d34f48ab32c47bfdd03cca P 1fa6b44ef46044d3a113266f1ebba9e4: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20260806 02:46:15.815153 10063 tablet_bootstrap.cc:492] T f1d165b802d34f48ab32c47bfdd03cca P 7d735936b19c4d7aa4b8db79c7694f1d: No bootstrap required, opened a new log
I20260806 02:46:15.815892 10063 ts_tablet_manager.cc:1397] T f1d165b802d34f48ab32c47bfdd03cca P 7d735936b19c4d7aa4b8db79c7694f1d: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.006s
I20260806 02:46:15.816798 10063 raft_consensus.cc:348] T f1d165b802d34f48ab32c47bfdd03cca P 7d735936b19c4d7aa4b8db79c7694f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } }
I20260806 02:46:15.817119 10063 raft_consensus.cc:374] T f1d165b802d34f48ab32c47bfdd03cca P 7d735936b19c4d7aa4b8db79c7694f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:15.817211 10063 raft_consensus.cc:729] T f1d165b802d34f48ab32c47bfdd03cca P 7d735936b19c4d7aa4b8db79c7694f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d735936b19c4d7aa4b8db79c7694f1d, State: Initialized, Role: FOLLOWER
I20260806 02:46:15.817531 10063 consensus_queue.cc:260] T f1d165b802d34f48ab32c47bfdd03cca P 7d735936b19c4d7aa4b8db79c7694f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } }
I20260806 02:46:15.818086 10070 tablet_replica.cc:439] T f1d165b802d34f48ab32c47bfdd03cca P 7d735936b19c4d7aa4b8db79c7694f1d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } } }
I20260806 02:46:15.818478 10070 tablet_replica.cc:442] T f1d165b802d34f48ab32c47bfdd03cca P 7d735936b19c4d7aa4b8db79c7694f1d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:46:15.819058 10063 ts_tablet_manager.cc:1428] T f1d165b802d34f48ab32c47bfdd03cca P 7d735936b19c4d7aa4b8db79c7694f1d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260806 02:46:15.890341 10070 raft_consensus.cc:482] T f1d165b802d34f48ab32c47bfdd03cca P 7d735936b19c4d7aa4b8db79c7694f1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:46:15.890682 10070 raft_consensus.cc:504] T f1d165b802d34f48ab32c47bfdd03cca P 7d735936b19c4d7aa4b8db79c7694f1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } }
I20260806 02:46:15.892504 10070 leader_election.cc:288] T f1d165b802d34f48ab32c47bfdd03cca P 7d735936b19c4d7aa4b8db79c7694f1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89153066de564aab9ff1c28037bb286c (127.1.210.66:37491), 1fa6b44ef46044d3a113266f1ebba9e4 (127.1.210.67:42521)
I20260806 02:46:15.904376 10059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:15.892991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:33911 (local address 127.1.210.66:37491)
0806 02:46:15.893218 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:15.893225 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:15.895706 (+ 2481us) server_negotiation.cc:407] Connection header received
0806 02:46:15.896265 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:15.896271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:15.896374 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:15.896443 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:46:15.897255 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.898428 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.901228 (+ 2800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.903173 (+ 1945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.903436 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:15.903553 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:15.903870 (+ 317us) server_negotiation.cc:299] Negotiation successful
0806 02:46:15.904034 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 02:46:15.904858 10103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:15.892712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:37491 (local address 127.1.210.65:33911)
0806 02:46:15.895534 (+ 2822us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:15.895558 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:15.896050 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:15.896682 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:15.896687 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:15.896714 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:46:15.897045 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:15.897058 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.898654 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.898659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:15.900974 (+ 2315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:15.900994 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.903418 (+ 2424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.903425 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:15.903661 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:15.903706 (+ 45us) client_negotiation.cc:769] Sending connection context
0806 02:46:15.903821 (+ 115us) client_negotiation.cc:240] Negotiation successful
0806 02:46:15.903984 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2697,"thread_start_us":11474,"threads_started":1}
I20260806 02:46:15.906054 9846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d165b802d34f48ab32c47bfdd03cca" candidate_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89153066de564aab9ff1c28037bb286c" is_pre_election: true
I20260806 02:46:15.906507 9846 raft_consensus.cc:2454] T f1d165b802d34f48ab32c47bfdd03cca P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d735936b19c4d7aa4b8db79c7694f1d in term 0.
I20260806 02:46:15.907196 9635 leader_election.cc:302] T f1d165b802d34f48ab32c47bfdd03cca P 7d735936b19c4d7aa4b8db79c7694f1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d735936b19c4d7aa4b8db79c7694f1d, 89153066de564aab9ff1c28037bb286c; no voters:
I20260806 02:46:15.907665 10070 raft_consensus.cc:2781] T f1d165b802d34f48ab32c47bfdd03cca P 7d735936b19c4d7aa4b8db79c7694f1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:46:15.907817 10070 raft_consensus.cc:482] T f1d165b802d34f48ab32c47bfdd03cca P 7d735936b19c4d7aa4b8db79c7694f1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:46:15.907948 10070 raft_consensus.cc:3037] T f1d165b802d34f48ab32c47bfdd03cca P 7d735936b19c4d7aa4b8db79c7694f1d [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:15.910295 10070 raft_consensus.cc:504] T f1d165b802d34f48ab32c47bfdd03cca P 7d735936b19c4d7aa4b8db79c7694f1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } }
I20260806 02:46:15.911166 10070 leader_election.cc:288] T f1d165b802d34f48ab32c47bfdd03cca P 7d735936b19c4d7aa4b8db79c7694f1d [CANDIDATE]: Term 1 election: Requested vote from peers 89153066de564aab9ff1c28037bb286c (127.1.210.66:37491), 1fa6b44ef46044d3a113266f1ebba9e4 (127.1.210.67:42521)
I20260806 02:46:15.911572 9846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d165b802d34f48ab32c47bfdd03cca" candidate_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89153066de564aab9ff1c28037bb286c"
I20260806 02:46:15.911803 9846 raft_consensus.cc:3037] T f1d165b802d34f48ab32c47bfdd03cca P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:15.912457 10104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:15.893361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:48437 (local address 127.1.210.67:42521)
0806 02:46:15.895188 (+ 1827us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:15.895195 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:15.904039 (+ 8844us) server_negotiation.cc:407] Connection header received
0806 02:46:15.904252 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:15.904258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:15.904337 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:15.904446 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:46:15.905732 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.906928 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.909681 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.911053 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.911251 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:15.911331 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:15.911956 (+ 625us) server_negotiation.cc:299] Negotiation successful
0806 02:46:15.912125 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1634,"thread_start_us":1516,"threads_started":1}
I20260806 02:46:15.913193 10102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:15.892921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:42521 (local address 127.1.210.65:48437)
0806 02:46:15.903918 (+ 10997us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:15.903940 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:15.904106 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:15.905101 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:15.905106 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:15.905134 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:46:15.905462 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:15.905472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.907055 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.907061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:15.909427 (+ 2366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:15.909447 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.911502 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.911511 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:15.911741 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:15.911779 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 02:46:15.912679 (+ 900us) client_negotiation.cc:240] Negotiation successful
0806 02:46:15.912844 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10823,"thread_start_us":2392,"threads_started":1}
I20260806 02:46:15.914390 9846 raft_consensus.cc:2454] T f1d165b802d34f48ab32c47bfdd03cca P 89153066de564aab9ff1c28037bb286c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d735936b19c4d7aa4b8db79c7694f1d in term 1.
I20260806 02:46:15.914989 9635 leader_election.cc:302] T f1d165b802d34f48ab32c47bfdd03cca P 7d735936b19c4d7aa4b8db79c7694f1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d735936b19c4d7aa4b8db79c7694f1d, 89153066de564aab9ff1c28037bb286c; no voters:
I20260806 02:46:15.915343 10070 raft_consensus.cc:2781] T f1d165b802d34f48ab32c47bfdd03cca P 7d735936b19c4d7aa4b8db79c7694f1d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:46:15.915535 9987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d165b802d34f48ab32c47bfdd03cca" candidate_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" is_pre_election: true
I20260806 02:46:15.915617 9986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d165b802d34f48ab32c47bfdd03cca" candidate_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fa6b44ef46044d3a113266f1ebba9e4"
I20260806 02:46:15.915880 9987 raft_consensus.cc:2454] T f1d165b802d34f48ab32c47bfdd03cca P 1fa6b44ef46044d3a113266f1ebba9e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d735936b19c4d7aa4b8db79c7694f1d in term 0.
I20260806 02:46:15.921039 10070 raft_consensus.cc:686] T f1d165b802d34f48ab32c47bfdd03cca P 7d735936b19c4d7aa4b8db79c7694f1d [term 1 LEADER]: Becoming Leader. State: Replica: 7d735936b19c4d7aa4b8db79c7694f1d, State: Running, Role: LEADER
I20260806 02:46:15.921532 10070 consensus_queue.cc:237] T f1d165b802d34f48ab32c47bfdd03cca P 7d735936b19c4d7aa4b8db79c7694f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } }
I20260806 02:46:15.924849 10105 tablet_replica.cc:439] T f1d165b802d34f48ab32c47bfdd03cca P 7d735936b19c4d7aa4b8db79c7694f1d: TxnStatusTablet state changed. Reason: New leader 7d735936b19c4d7aa4b8db79c7694f1d. Latest consensus state: current_term: 1 leader_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } } }
I20260806 02:46:15.925283 10105 tablet_replica.cc:442] T f1d165b802d34f48ab32c47bfdd03cca P 7d735936b19c4d7aa4b8db79c7694f1d: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:46:15.925765 9542 catalog_manager.cc:5495] T f1d165b802d34f48ab32c47bfdd03cca P 7d735936b19c4d7aa4b8db79c7694f1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d735936b19c4d7aa4b8db79c7694f1d (127.1.210.65). New cstate: current_term: 1 leader_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } health_report { overall_health: UNKNOWN } } }
I20260806 02:46:15.926136 10107 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 02:46:15.926400 10107 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 02:46:15.994055 10097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:15.970756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:33721 (local address 127.0.0.1:38882)
0806 02:46:15.971263 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:15.971292 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:15.971494 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:15.987908 (+ 16414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:15.987914 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:15.987943 (+ 29us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:46:15.988222 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:15.988238 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.990042 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.990047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:15.991216 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:15.991232 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.992090 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.992098 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:15.992268 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:15.992342 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:46:15.993393 (+ 1051us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:46:15.993447 (+ 54us) client_negotiation.cc:769] Sending connection context
0806 02:46:15.993563 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 02:46:15.993735 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260}
I20260806 02:46:15.994127 10110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:15.970762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38882 (local address 127.1.210.65:33721)
0806 02:46:15.987263 (+ 16501us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:15.987269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:15.987326 (+ 57us) server_negotiation.cc:407] Connection header received
0806 02:46:15.987455 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:15.987461 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:15.987582 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:15.988413 (+ 831us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:46:15.988713 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.989838 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.991405 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.991881 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.992041 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:15.992585 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:46:15.993278 (+ 693us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:46:15.993436 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:15.993587 (+ 151us) server_negotiation.cc:299] Negotiation successful
0806 02:46:15.993679 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16345,"thread_start_us":16139,"threads_started":1}
I20260806 02:46:16.001856 9987 raft_consensus.cc:3037] T f1d165b802d34f48ab32c47bfdd03cca P 1fa6b44ef46044d3a113266f1ebba9e4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:16.002554 9846 raft_consensus.cc:1261] T f1d165b802d34f48ab32c47bfdd03cca P 89153066de564aab9ff1c28037bb286c [term 1 FOLLOWER]: Refusing update from remote peer 7d735936b19c4d7aa4b8db79c7694f1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:46:16.003665 10105 consensus_queue.cc:1035] T f1d165b802d34f48ab32c47bfdd03cca P 7d735936b19c4d7aa4b8db79c7694f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:46:16.005613 9987 raft_consensus.cc:1261] T f1d165b802d34f48ab32c47bfdd03cca P 1fa6b44ef46044d3a113266f1ebba9e4 [term 1 FOLLOWER]: Refusing update from remote peer 7d735936b19c4d7aa4b8db79c7694f1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:46:16.007565 10105 consensus_queue.cc:1035] T f1d165b802d34f48ab32c47bfdd03cca P 7d735936b19c4d7aa4b8db79c7694f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:46:16.019075 10068 tablet_replica.cc:439] T f1d165b802d34f48ab32c47bfdd03cca P 89153066de564aab9ff1c28037bb286c: TxnStatusTablet state changed. Reason: New leader 7d735936b19c4d7aa4b8db79c7694f1d. Latest consensus state: current_term: 1 leader_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } } }
I20260806 02:46:16.019582 10068 tablet_replica.cc:442] T f1d165b802d34f48ab32c47bfdd03cca P 89153066de564aab9ff1c28037bb286c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:46:16.025969 10070 tablet_replica.cc:439] T f1d165b802d34f48ab32c47bfdd03cca P 7d735936b19c4d7aa4b8db79c7694f1d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } } }
I20260806 02:46:16.026372 10070 tablet_replica.cc:442] T f1d165b802d34f48ab32c47bfdd03cca P 7d735936b19c4d7aa4b8db79c7694f1d: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:46:16.027001 10112 mvcc.cc:205] Tried to move back new op lower bound from 7315392004094652416 to 7315392003778854912. Current Snapshot: MvccSnapshot[applied={T|T < 7315392004094652416 or (T in {7315392004094652416})}]
I20260806 02:46:16.027933 10068 tablet_replica.cc:439] T f1d165b802d34f48ab32c47bfdd03cca P 89153066de564aab9ff1c28037bb286c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } } }
I20260806 02:46:16.028218 10068 tablet_replica.cc:442] T f1d165b802d34f48ab32c47bfdd03cca P 89153066de564aab9ff1c28037bb286c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:46:16.031137 10069 tablet_replica.cc:439] T f1d165b802d34f48ab32c47bfdd03cca P 1fa6b44ef46044d3a113266f1ebba9e4: TxnStatusTablet state changed. Reason: New leader 7d735936b19c4d7aa4b8db79c7694f1d. Latest consensus state: current_term: 1 leader_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } } }
I20260806 02:46:16.031497 10069 tablet_replica.cc:442] T f1d165b802d34f48ab32c47bfdd03cca P 1fa6b44ef46044d3a113266f1ebba9e4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:46:16.032845 10105 tablet_replica.cc:439] T f1d165b802d34f48ab32c47bfdd03cca P 7d735936b19c4d7aa4b8db79c7694f1d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } } }
I20260806 02:46:16.033138 10105 tablet_replica.cc:442] T f1d165b802d34f48ab32c47bfdd03cca P 7d735936b19c4d7aa4b8db79c7694f1d: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:46:16.035537 10069 tablet_replica.cc:439] T f1d165b802d34f48ab32c47bfdd03cca P 1fa6b44ef46044d3a113266f1ebba9e4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } } }
I20260806 02:46:16.035802 10069 tablet_replica.cc:442] T f1d165b802d34f48ab32c47bfdd03cca P 1fa6b44ef46044d3a113266f1ebba9e4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:46:16.053892 10110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:16.046404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:41491 (local address 127.1.210.65:33721)
0806 02:46:16.046623 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:16.046631 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:16.046656 (+ 25us) server_negotiation.cc:407] Connection header received
0806 02:46:16.046832 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:16.046838 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:16.046940 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:16.047053 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:46:16.047894 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:16.049299 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:16.051523 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:16.052254 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:16.052411 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:16.052936 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:46:16.053119 (+ 183us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:46:16.053215 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:16.053472 (+ 257us) server_negotiation.cc:299] Negotiation successful
0806 02:46:16.053567 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 02:46:16.053946 10121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:16.045671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:33721 (local address 127.1.210.66:41491)
0806 02:46:16.046465 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:16.046488 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:16.046664 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:16.047249 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:16.047253 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:16.047264 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:46:16.047697 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:16.047708 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:16.049541 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:16.049546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:16.051258 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:16.051278 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:16.052505 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:16.052511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:16.052656 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:16.052734 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:46:16.053297 (+ 563us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:46:16.053309 (+ 12us) client_negotiation.cc:769] Sending connection context
0806 02:46:16.053450 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 02:46:16.053586 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":799,"threads_started":1}
I20260806 02:46:16.065348 9704 raft_consensus.cc:1261] T b6ba3916e6f94e8184289fc2323f520d P 7d735936b19c4d7aa4b8db79c7694f1d [term 1 FOLLOWER]: Refusing update from remote peer 89153066de564aab9ff1c28037bb286c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:46:16.066057 10073 consensus_queue.cc:1035] T b6ba3916e6f94e8184289fc2323f520d P 89153066de564aab9ff1c28037bb286c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:46:16.066190 9987 raft_consensus.cc:1261] T b6ba3916e6f94e8184289fc2323f520d P 1fa6b44ef46044d3a113266f1ebba9e4 [term 1 FOLLOWER]: Refusing update from remote peer 89153066de564aab9ff1c28037bb286c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:46:16.066932 10073 consensus_queue.cc:1035] T b6ba3916e6f94e8184289fc2323f520d P 89153066de564aab9ff1c28037bb286c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:46:16.074290 10111 mvcc.cc:205] Tried to move back new op lower bound from 7315392004357177344 to 7315392002827210752. Current Snapshot: MvccSnapshot[applied={T|T < 7315392004357177344}]
I20260806 02:46:16.122797 10059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:16.108292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:58463 (local address 127.1.210.66:37491)
0806 02:46:16.108518 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:16.108526 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:16.115503 (+ 6977us) server_negotiation.cc:407] Connection header received
0806 02:46:16.115752 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:16.115758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:16.115865 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:16.115972 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:46:16.117018 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:16.118170 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:16.120552 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:16.121086 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:16.121261 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:16.121854 (+ 593us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:46:16.122009 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:46:16.122114 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:16.122330 (+ 216us) server_negotiation.cc:299] Negotiation successful
0806 02:46:16.122423 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 02:46:16.123023 10126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:16.108620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:37491 (local address 127.1.210.65:58463)
0806 02:46:16.115368 (+ 6748us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:16.115387 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:16.115596 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:16.116283 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:16.116288 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:16.116303 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:46:16.116777 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:16.116793 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:16.118316 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:16.118323 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:16.120347 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:16.120366 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:16.121323 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:16.121333 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:16.121528 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:16.121630 (+ 102us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:46:16.122142 (+ 512us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:46:16.122155 (+ 13us) client_negotiation.cc:769] Sending connection context
0806 02:46:16.122295 (+ 140us) client_negotiation.cc:240] Negotiation successful
0806 02:46:16.122511 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6528,"thread_start_us":6344,"threads_started":1}
I20260806 02:46:16.166260 9544 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:46:16.166581 9544 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:46:16.170022 9544 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e49507f75394515a4d7645a0f7848c1: Sending DeleteTablet for 3 replicas of tablet b6ba3916e6f94e8184289fc2323f520d
I20260806 02:46:16.171000 9825 tablet_service.cc:1514] Processing DeleteTablet for tablet b6ba3916e6f94e8184289fc2323f520d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:46:16 UTC) from {username='oryx', principal='oryx/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:53318
I20260806 02:46:16.171389 9966 tablet_service.cc:1514] Processing DeleteTablet for tablet b6ba3916e6f94e8184289fc2323f520d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:46:16 UTC) from {username='oryx', principal='oryx/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:55050
I20260806 02:46:16.171908 9683 tablet_service.cc:1514] Processing DeleteTablet for tablet b6ba3916e6f94e8184289fc2323f520d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:46:16 UTC) from {username='oryx', principal='oryx/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:38874
I20260806 02:46:16.172107 10130 tablet_replica.cc:330] T b6ba3916e6f94e8184289fc2323f520d P 1fa6b44ef46044d3a113266f1ebba9e4: stopping tablet replica
I20260806 02:46:16.172319 10130 raft_consensus.cc:2229] T b6ba3916e6f94e8184289fc2323f520d P 1fa6b44ef46044d3a113266f1ebba9e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:46:16.172583 10130 raft_consensus.cc:2258] T b6ba3916e6f94e8184289fc2323f520d P 1fa6b44ef46044d3a113266f1ebba9e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:46:16.173779 10130 ts_tablet_manager.cc:1905] T b6ba3916e6f94e8184289fc2323f520d P 1fa6b44ef46044d3a113266f1ebba9e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:46:16.181805 10130 ts_tablet_manager.cc:1918] T b6ba3916e6f94e8184289fc2323f520d P 1fa6b44ef46044d3a113266f1ebba9e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:46:16.181994 10130 log.cc:1196] T b6ba3916e6f94e8184289fc2323f520d P 1fa6b44ef46044d3a113266f1ebba9e4: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-2/wal/wals/b6ba3916e6f94e8184289fc2323f520d
I20260806 02:46:16.182487 10129 tablet_replica.cc:330] T b6ba3916e6f94e8184289fc2323f520d P 89153066de564aab9ff1c28037bb286c: stopping tablet replica
I20260806 02:46:16.182512 10130 ts_tablet_manager.cc:1939] T b6ba3916e6f94e8184289fc2323f520d P 1fa6b44ef46044d3a113266f1ebba9e4: Deleting consensus metadata
I20260806 02:46:16.182858 10131 tablet_replica.cc:330] T b6ba3916e6f94e8184289fc2323f520d P 7d735936b19c4d7aa4b8db79c7694f1d: stopping tablet replica
I20260806 02:46:16.182904 10129 raft_consensus.cc:2229] T b6ba3916e6f94e8184289fc2323f520d P 89153066de564aab9ff1c28037bb286c [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:46:16.183225 10131 raft_consensus.cc:2229] T b6ba3916e6f94e8184289fc2323f520d P 7d735936b19c4d7aa4b8db79c7694f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:46:16.183544 10131 raft_consensus.cc:2258] T b6ba3916e6f94e8184289fc2323f520d P 7d735936b19c4d7aa4b8db79c7694f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:46:16.184195 9528 catalog_manager.cc:4841] TS 1fa6b44ef46044d3a113266f1ebba9e4 (127.1.210.67:42521): tablet b6ba3916e6f94e8184289fc2323f520d (table test-table [id=3e2ab77cf7454a60958abf6b6a9a718c]) successfully deleted
I20260806 02:46:16.183537 10129 raft_consensus.cc:2258] T b6ba3916e6f94e8184289fc2323f520d P 89153066de564aab9ff1c28037bb286c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:46:16.185022 10131 ts_tablet_manager.cc:1905] T b6ba3916e6f94e8184289fc2323f520d P 7d735936b19c4d7aa4b8db79c7694f1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:46:16.186012 10129 ts_tablet_manager.cc:1905] T b6ba3916e6f94e8184289fc2323f520d P 89153066de564aab9ff1c28037bb286c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:46:16.193648 10129 ts_tablet_manager.cc:1918] T b6ba3916e6f94e8184289fc2323f520d P 89153066de564aab9ff1c28037bb286c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:46:16.193902 10129 log.cc:1196] T b6ba3916e6f94e8184289fc2323f520d P 89153066de564aab9ff1c28037bb286c: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-1/wal/wals/b6ba3916e6f94e8184289fc2323f520d
I20260806 02:46:16.194098 10131 ts_tablet_manager.cc:1918] T b6ba3916e6f94e8184289fc2323f520d P 7d735936b19c4d7aa4b8db79c7694f1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:46:16.194267 10131 log.cc:1196] T b6ba3916e6f94e8184289fc2323f520d P 7d735936b19c4d7aa4b8db79c7694f1d: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-0/wal/wals/b6ba3916e6f94e8184289fc2323f520d
I20260806 02:46:16.194603 10129 ts_tablet_manager.cc:1939] T b6ba3916e6f94e8184289fc2323f520d P 89153066de564aab9ff1c28037bb286c: Deleting consensus metadata
I20260806 02:46:16.194837 10131 ts_tablet_manager.cc:1939] T b6ba3916e6f94e8184289fc2323f520d P 7d735936b19c4d7aa4b8db79c7694f1d: Deleting consensus metadata
I20260806 02:46:16.196022 9530 catalog_manager.cc:4841] TS 89153066de564aab9ff1c28037bb286c (127.1.210.66:37491): tablet b6ba3916e6f94e8184289fc2323f520d (table test-table [id=3e2ab77cf7454a60958abf6b6a9a718c]) successfully deleted
I20260806 02:46:16.196192 9528 catalog_manager.cc:4841] TS 7d735936b19c4d7aa4b8db79c7694f1d (127.1.210.65:33721): tablet b6ba3916e6f94e8184289fc2323f520d (table test-table [id=3e2ab77cf7454a60958abf6b6a9a718c]) successfully deleted
Aug 06 02:46:16 dist-test-slave-8tdl krb5kdc[9499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 02:46:16 dist-test-slave-8tdl krb5kdc[9499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:16 dist-test-slave-8tdl krb5kdc[9499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
W20260806 02:46:16.247146 10098 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:16.232881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33740 (local address 127.1.210.126:36511)
0806 02:46:16.233160 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:16.233167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:16.233570 (+ 403us) server_negotiation.cc:407] Connection header received
0806 02:46:16.233819 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:16.233823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:16.233895 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:16.234005 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:16.235953 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:16.236765 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:16.238761 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:16.239519 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:16.239652 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:16.246885 (+ 7233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33740: BlockingRecv error: recv got EOF from 127.0.0.1:33740 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 06 02:46:16 dist-test-slave-8tdl krb5kdc[9499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM
I20260806 02:46:16.267678 10098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:16.251878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33750 (local address 127.1.210.126:36511)
0806 02:46:16.252166 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:16.252174 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:16.252448 (+ 274us) server_negotiation.cc:407] Connection header received
0806 02:46:16.252682 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:16.252687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:16.252774 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:16.252883 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:16.254192 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:16.255075 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:16.256396 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:16.256847 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:16.257005 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:16.261199 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:16.261204 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:16.261208 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:16.261251 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:16.264439 (+ 3188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:16.265294 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:16.265299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:16.265301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:16.265460 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:16.265899 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:16.265906 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:16.265914 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:16.266320 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:16.266634 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:16.267044 (+ 410us) server_negotiation.cc:299] Negotiation successful
0806 02:46:16.267306 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260806 02:46:16.272315 9544 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:46:16.273105 9544 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:46:16.281570 9683 tablet_service.cc:1467] Processing CreateTablet for tablet 9f9aa52d89ab4a5bad452e82aaf8b775 (DEFAULT_TABLE table=test-table [id=e77d050033854e61824e7a355f5fe577]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:46:16.281844 9825 tablet_service.cc:1467] Processing CreateTablet for tablet 9f9aa52d89ab4a5bad452e82aaf8b775 (DEFAULT_TABLE table=test-table [id=e77d050033854e61824e7a355f5fe577]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:46:16.282147 9683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f9aa52d89ab4a5bad452e82aaf8b775. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:16.282250 9825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f9aa52d89ab4a5bad452e82aaf8b775. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:16.288093 9966 tablet_service.cc:1467] Processing CreateTablet for tablet 9f9aa52d89ab4a5bad452e82aaf8b775 (DEFAULT_TABLE table=test-table [id=e77d050033854e61824e7a355f5fe577]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:46:16.288692 9966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f9aa52d89ab4a5bad452e82aaf8b775. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:16.289873 10062 tablet_bootstrap.cc:492] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 89153066de564aab9ff1c28037bb286c: Bootstrap starting.
I20260806 02:46:16.289883 10063 tablet_bootstrap.cc:492] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 7d735936b19c4d7aa4b8db79c7694f1d: Bootstrap starting.
I20260806 02:46:16.291693 10062 tablet_bootstrap.cc:654] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 89153066de564aab9ff1c28037bb286c: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:16.291706 10063 tablet_bootstrap.cc:654] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 7d735936b19c4d7aa4b8db79c7694f1d: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:16.296823 10064 tablet_bootstrap.cc:492] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 1fa6b44ef46044d3a113266f1ebba9e4: Bootstrap starting.
I20260806 02:46:16.298605 10064 tablet_bootstrap.cc:654] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 1fa6b44ef46044d3a113266f1ebba9e4: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:16.303052 10062 tablet_bootstrap.cc:492] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 89153066de564aab9ff1c28037bb286c: No bootstrap required, opened a new log
I20260806 02:46:16.303288 10062 ts_tablet_manager.cc:1397] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 89153066de564aab9ff1c28037bb286c: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.001s
I20260806 02:46:16.304136 10064 tablet_bootstrap.cc:492] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 1fa6b44ef46044d3a113266f1ebba9e4: No bootstrap required, opened a new log
I20260806 02:46:16.304085 10062 raft_consensus.cc:348] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } }
I20260806 02:46:16.304296 10064 ts_tablet_manager.cc:1397] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 1fa6b44ef46044d3a113266f1ebba9e4: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260806 02:46:16.304365 10062 raft_consensus.cc:374] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:16.304463 10062 raft_consensus.cc:729] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89153066de564aab9ff1c28037bb286c, State: Initialized, Role: FOLLOWER
I20260806 02:46:16.304700 10062 consensus_queue.cc:260] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 89153066de564aab9ff1c28037bb286c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } }
I20260806 02:46:16.305181 10062 ts_tablet_manager.cc:1428] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 89153066de564aab9ff1c28037bb286c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 02:46:16.305083 10064 raft_consensus.cc:348] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 1fa6b44ef46044d3a113266f1ebba9e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } }
I20260806 02:46:16.305351 10064 raft_consensus.cc:374] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 1fa6b44ef46044d3a113266f1ebba9e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:16.305440 10064 raft_consensus.cc:729] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 1fa6b44ef46044d3a113266f1ebba9e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fa6b44ef46044d3a113266f1ebba9e4, State: Initialized, Role: FOLLOWER
I20260806 02:46:16.305816 10064 consensus_queue.cc:260] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 1fa6b44ef46044d3a113266f1ebba9e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } }
I20260806 02:46:16.306223 10063 tablet_bootstrap.cc:492] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 7d735936b19c4d7aa4b8db79c7694f1d: No bootstrap required, opened a new log
I20260806 02:46:16.306416 10063 ts_tablet_manager.cc:1397] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 7d735936b19c4d7aa4b8db79c7694f1d: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.008s
I20260806 02:46:16.307227 10063 raft_consensus.cc:348] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 7d735936b19c4d7aa4b8db79c7694f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } }
I20260806 02:46:16.307483 10063 raft_consensus.cc:374] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 7d735936b19c4d7aa4b8db79c7694f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:16.307588 10063 raft_consensus.cc:729] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 7d735936b19c4d7aa4b8db79c7694f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d735936b19c4d7aa4b8db79c7694f1d, State: Initialized, Role: FOLLOWER
I20260806 02:46:16.307859 10063 consensus_queue.cc:260] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 7d735936b19c4d7aa4b8db79c7694f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } }
I20260806 02:46:16.310509 10063 ts_tablet_manager.cc:1428] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 7d735936b19c4d7aa4b8db79c7694f1d: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260806 02:46:16.311286 10064 ts_tablet_manager.cc:1428] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 1fa6b44ef46044d3a113266f1ebba9e4: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260806 02:46:16.439390 10068 raft_consensus.cc:482] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:46:16.439683 10068 raft_consensus.cc:504] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } }
I20260806 02:46:16.440498 10068 leader_election.cc:288] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 89153066de564aab9ff1c28037bb286c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d735936b19c4d7aa4b8db79c7694f1d (127.1.210.65:33721), 1fa6b44ef46044d3a113266f1ebba9e4 (127.1.210.67:42521)
I20260806 02:46:16.441109 9704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f9aa52d89ab4a5bad452e82aaf8b775" candidate_uuid: "89153066de564aab9ff1c28037bb286c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" is_pre_election: true
I20260806 02:46:16.441147 9987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f9aa52d89ab4a5bad452e82aaf8b775" candidate_uuid: "89153066de564aab9ff1c28037bb286c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" is_pre_election: true
I20260806 02:46:16.441356 9704 raft_consensus.cc:2454] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 7d735936b19c4d7aa4b8db79c7694f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89153066de564aab9ff1c28037bb286c in term 0.
I20260806 02:46:16.441464 9987 raft_consensus.cc:2454] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 1fa6b44ef46044d3a113266f1ebba9e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89153066de564aab9ff1c28037bb286c in term 0.
I20260806 02:46:16.441953 9775 leader_election.cc:302] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 89153066de564aab9ff1c28037bb286c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d735936b19c4d7aa4b8db79c7694f1d, 89153066de564aab9ff1c28037bb286c; no voters:
I20260806 02:46:16.442371 10068 raft_consensus.cc:2781] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:46:16.442504 10068 raft_consensus.cc:482] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:46:16.442653 10068 raft_consensus.cc:3037] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:16.445053 10068 raft_consensus.cc:504] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 89153066de564aab9ff1c28037bb286c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } }
I20260806 02:46:16.445667 10068 leader_election.cc:288] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 89153066de564aab9ff1c28037bb286c [CANDIDATE]: Term 1 election: Requested vote from peers 7d735936b19c4d7aa4b8db79c7694f1d (127.1.210.65:33721), 1fa6b44ef46044d3a113266f1ebba9e4 (127.1.210.67:42521)
I20260806 02:46:16.446345 9987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f9aa52d89ab4a5bad452e82aaf8b775" candidate_uuid: "89153066de564aab9ff1c28037bb286c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fa6b44ef46044d3a113266f1ebba9e4"
I20260806 02:46:16.446345 9704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f9aa52d89ab4a5bad452e82aaf8b775" candidate_uuid: "89153066de564aab9ff1c28037bb286c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d735936b19c4d7aa4b8db79c7694f1d"
I20260806 02:46:16.446579 9987 raft_consensus.cc:3037] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 1fa6b44ef46044d3a113266f1ebba9e4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:16.446586 9704 raft_consensus.cc:3037] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 7d735936b19c4d7aa4b8db79c7694f1d [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:16.449671 9704 raft_consensus.cc:2454] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 7d735936b19c4d7aa4b8db79c7694f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89153066de564aab9ff1c28037bb286c in term 1.
I20260806 02:46:16.449671 9987 raft_consensus.cc:2454] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 1fa6b44ef46044d3a113266f1ebba9e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89153066de564aab9ff1c28037bb286c in term 1.
I20260806 02:46:16.450206 9775 leader_election.cc:302] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 89153066de564aab9ff1c28037bb286c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fa6b44ef46044d3a113266f1ebba9e4, 89153066de564aab9ff1c28037bb286c; no voters:
I20260806 02:46:16.450616 10068 raft_consensus.cc:2781] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 89153066de564aab9ff1c28037bb286c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:46:16.450866 10068 raft_consensus.cc:686] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 89153066de564aab9ff1c28037bb286c [term 1 LEADER]: Becoming Leader. State: Replica: 89153066de564aab9ff1c28037bb286c, State: Running, Role: LEADER
I20260806 02:46:16.451167 10068 consensus_queue.cc:237] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 89153066de564aab9ff1c28037bb286c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } }
I20260806 02:46:16.453895 9543 catalog_manager.cc:5495] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 89153066de564aab9ff1c28037bb286c reported cstate change: term changed from 0 to 1, leader changed from <none> to 89153066de564aab9ff1c28037bb286c (127.1.210.66). New cstate: current_term: 1 leader_uuid: "89153066de564aab9ff1c28037bb286c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } health_report { overall_health: UNKNOWN } } }
I20260806 02:46:16.470966 10059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:16.466048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53334 (local address 127.1.210.66:37491)
0806 02:46:16.466260 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:16.466265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:16.466282 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:46:16.466342 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:16.466346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:16.466410 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:16.466495 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:46:16.466998 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:16.467767 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:16.469138 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:16.469637 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:16.469758 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:16.470188 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:46:16.470311 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:46:16.470388 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:16.470516 (+ 128us) server_negotiation.cc:299] Negotiation successful
0806 02:46:16.470621 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 02:46:16.473829 9987 raft_consensus.cc:1261] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 1fa6b44ef46044d3a113266f1ebba9e4 [term 1 FOLLOWER]: Refusing update from remote peer 89153066de564aab9ff1c28037bb286c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:46:16.473878 9704 raft_consensus.cc:1261] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 7d735936b19c4d7aa4b8db79c7694f1d [term 1 FOLLOWER]: Refusing update from remote peer 89153066de564aab9ff1c28037bb286c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:46:16.474550 10128 consensus_queue.cc:1035] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 89153066de564aab9ff1c28037bb286c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:46:16.475008 10068 consensus_queue.cc:1035] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 89153066de564aab9ff1c28037bb286c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:46:16.487550 9542 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:46:16.487820 9542 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:46:16.491822 9542 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e49507f75394515a4d7645a0f7848c1: Sending DeleteTablet for 3 replicas of tablet 9f9aa52d89ab4a5bad452e82aaf8b775
I20260806 02:46:16.493048 9825 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f9aa52d89ab4a5bad452e82aaf8b775 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:46:16 UTC) from {username='oryx', principal='oryx/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:53318
I20260806 02:46:16.493077 9966 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f9aa52d89ab4a5bad452e82aaf8b775 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:46:16 UTC) from {username='oryx', principal='oryx/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:55050
I20260806 02:46:16.493394 10130 tablet_replica.cc:330] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 1fa6b44ef46044d3a113266f1ebba9e4: stopping tablet replica
I20260806 02:46:16.493412 9824 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f9aa52d89ab4a5bad452e82aaf8b775 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:46:16 UTC) from {username='oryx', principal='oryx/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:53318
I20260806 02:46:16.493624 10129 tablet_replica.cc:330] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 89153066de564aab9ff1c28037bb286c: stopping tablet replica
I20260806 02:46:16.493637 10130 raft_consensus.cc:2229] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 1fa6b44ef46044d3a113266f1ebba9e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:46:16.493804 10129 raft_consensus.cc:2229] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 89153066de564aab9ff1c28037bb286c [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:46:16.493889 10130 raft_consensus.cc:2258] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 1fa6b44ef46044d3a113266f1ebba9e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:46:16.494154 10129 raft_consensus.cc:2258] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 89153066de564aab9ff1c28037bb286c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:46:16.495393 10129 ts_tablet_manager.cc:1905] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 89153066de564aab9ff1c28037bb286c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:46:16.495428 10130 ts_tablet_manager.cc:1905] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 1fa6b44ef46044d3a113266f1ebba9e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:46:16.495466 9544 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:46:16.496176 9544 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:46:16.500574 10130 ts_tablet_manager.cc:1918] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 1fa6b44ef46044d3a113266f1ebba9e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:46:16.500748 10130 log.cc:1196] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 1fa6b44ef46044d3a113266f1ebba9e4: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-2/wal/wals/9f9aa52d89ab4a5bad452e82aaf8b775
I20260806 02:46:16.501235 10130 ts_tablet_manager.cc:1939] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 1fa6b44ef46044d3a113266f1ebba9e4: Deleting consensus metadata
I20260806 02:46:16.501900 10129 ts_tablet_manager.cc:1918] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 89153066de564aab9ff1c28037bb286c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:46:16.502077 10129 log.cc:1196] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 89153066de564aab9ff1c28037bb286c: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-1/wal/wals/9f9aa52d89ab4a5bad452e82aaf8b775
I20260806 02:46:16.502417 9528 catalog_manager.cc:4841] TS 1fa6b44ef46044d3a113266f1ebba9e4 (127.1.210.67:42521): tablet 9f9aa52d89ab4a5bad452e82aaf8b775 (table test-table [id=e77d050033854e61824e7a355f5fe577]) successfully deleted
I20260806 02:46:16.502501 10129 ts_tablet_manager.cc:1939] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 89153066de564aab9ff1c28037bb286c: Deleting consensus metadata
I20260806 02:46:16.503618 9530 catalog_manager.cc:4841] TS 89153066de564aab9ff1c28037bb286c (127.1.210.66:37491): tablet 9f9aa52d89ab4a5bad452e82aaf8b775 (table test-table [id=e77d050033854e61824e7a355f5fe577]) successfully deleted
W20260806 02:46:16.504222 9530 catalog_manager.cc:4808] TS 89153066de564aab9ff1c28037bb286c (127.1.210.66:37491): delete failed for tablet 9f9aa52d89ab4a5bad452e82aaf8b775 because the tablet was not found. No further retry: Not found: Tablet not found: 9f9aa52d89ab4a5bad452e82aaf8b775
I20260806 02:46:16.504504 9683 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f9aa52d89ab4a5bad452e82aaf8b775 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:46:16 UTC) from {username='oryx', principal='oryx/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:38874
I20260806 02:46:16.504748 9966 tablet_service.cc:1467] Processing CreateTablet for tablet 1d460a7404b54944ae05e39d92f94273 (DEFAULT_TABLE table=test-table [id=601bd31414c9401aaa01c8259a6c9449]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:46:16.504814 10131 tablet_replica.cc:330] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 7d735936b19c4d7aa4b8db79c7694f1d: stopping tablet replica
I20260806 02:46:16.505066 9966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d460a7404b54944ae05e39d92f94273. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:16.505028 10131 raft_consensus.cc:2229] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 7d735936b19c4d7aa4b8db79c7694f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:46:16.505074 9682 tablet_service.cc:1467] Processing CreateTablet for tablet 1d460a7404b54944ae05e39d92f94273 (DEFAULT_TABLE table=test-table [id=601bd31414c9401aaa01c8259a6c9449]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:46:16.505270 10131 raft_consensus.cc:2258] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 7d735936b19c4d7aa4b8db79c7694f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:46:16.505447 9682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d460a7404b54944ae05e39d92f94273. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:16.506357 10131 ts_tablet_manager.cc:1905] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 7d735936b19c4d7aa4b8db79c7694f1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:46:16.510377 10064 tablet_bootstrap.cc:492] T 1d460a7404b54944ae05e39d92f94273 P 1fa6b44ef46044d3a113266f1ebba9e4: Bootstrap starting.
I20260806 02:46:16.511101 9824 tablet_service.cc:1467] Processing CreateTablet for tablet 1d460a7404b54944ae05e39d92f94273 (DEFAULT_TABLE table=test-table [id=601bd31414c9401aaa01c8259a6c9449]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:46:16.511557 9824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d460a7404b54944ae05e39d92f94273. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:16.512017 10064 tablet_bootstrap.cc:654] T 1d460a7404b54944ae05e39d92f94273 P 1fa6b44ef46044d3a113266f1ebba9e4: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:16.512765 10131 ts_tablet_manager.cc:1918] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 7d735936b19c4d7aa4b8db79c7694f1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:46:16.512959 10131 log.cc:1196] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 7d735936b19c4d7aa4b8db79c7694f1d: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-0/wal/wals/9f9aa52d89ab4a5bad452e82aaf8b775
I20260806 02:46:16.513527 10131 ts_tablet_manager.cc:1939] T 9f9aa52d89ab4a5bad452e82aaf8b775 P 7d735936b19c4d7aa4b8db79c7694f1d: Deleting consensus metadata
I20260806 02:46:16.514901 9528 catalog_manager.cc:4841] TS 7d735936b19c4d7aa4b8db79c7694f1d (127.1.210.65:33721): tablet 9f9aa52d89ab4a5bad452e82aaf8b775 (table test-table [id=e77d050033854e61824e7a355f5fe577]) successfully deleted
I20260806 02:46:16.517347 10062 tablet_bootstrap.cc:492] T 1d460a7404b54944ae05e39d92f94273 P 89153066de564aab9ff1c28037bb286c: Bootstrap starting.
I20260806 02:46:16.519119 10062 tablet_bootstrap.cc:654] T 1d460a7404b54944ae05e39d92f94273 P 89153066de564aab9ff1c28037bb286c: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:16.519970 10063 tablet_bootstrap.cc:492] T 1d460a7404b54944ae05e39d92f94273 P 7d735936b19c4d7aa4b8db79c7694f1d: Bootstrap starting.
I20260806 02:46:16.521754 10062 tablet_bootstrap.cc:492] T 1d460a7404b54944ae05e39d92f94273 P 89153066de564aab9ff1c28037bb286c: No bootstrap required, opened a new log
I20260806 02:46:16.521924 10062 ts_tablet_manager.cc:1397] T 1d460a7404b54944ae05e39d92f94273 P 89153066de564aab9ff1c28037bb286c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 02:46:16.521951 10063 tablet_bootstrap.cc:654] T 1d460a7404b54944ae05e39d92f94273 P 7d735936b19c4d7aa4b8db79c7694f1d: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:16.522562 10064 tablet_bootstrap.cc:492] T 1d460a7404b54944ae05e39d92f94273 P 1fa6b44ef46044d3a113266f1ebba9e4: No bootstrap required, opened a new log
I20260806 02:46:16.522781 10064 ts_tablet_manager.cc:1397] T 1d460a7404b54944ae05e39d92f94273 P 1fa6b44ef46044d3a113266f1ebba9e4: Time spent bootstrapping tablet: real 0.012s user 0.012s sys 0.000s
I20260806 02:46:16.522738 10062 raft_consensus.cc:348] T 1d460a7404b54944ae05e39d92f94273 P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } }
I20260806 02:46:16.523031 10062 raft_consensus.cc:374] T 1d460a7404b54944ae05e39d92f94273 P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:16.523123 10062 raft_consensus.cc:729] T 1d460a7404b54944ae05e39d92f94273 P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89153066de564aab9ff1c28037bb286c, State: Initialized, Role: FOLLOWER
I20260806 02:46:16.523419 10062 consensus_queue.cc:260] T 1d460a7404b54944ae05e39d92f94273 P 89153066de564aab9ff1c28037bb286c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } }
I20260806 02:46:16.523727 10064 raft_consensus.cc:348] T 1d460a7404b54944ae05e39d92f94273 P 1fa6b44ef46044d3a113266f1ebba9e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } }
I20260806 02:46:16.524017 10064 raft_consensus.cc:374] T 1d460a7404b54944ae05e39d92f94273 P 1fa6b44ef46044d3a113266f1ebba9e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:16.524044 10062 ts_tablet_manager.cc:1428] T 1d460a7404b54944ae05e39d92f94273 P 89153066de564aab9ff1c28037bb286c: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260806 02:46:16.524116 10064 raft_consensus.cc:729] T 1d460a7404b54944ae05e39d92f94273 P 1fa6b44ef46044d3a113266f1ebba9e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fa6b44ef46044d3a113266f1ebba9e4, State: Initialized, Role: FOLLOWER
I20260806 02:46:16.524693 10063 tablet_bootstrap.cc:492] T 1d460a7404b54944ae05e39d92f94273 P 7d735936b19c4d7aa4b8db79c7694f1d: No bootstrap required, opened a new log
I20260806 02:46:16.524542 10064 consensus_queue.cc:260] T 1d460a7404b54944ae05e39d92f94273 P 1fa6b44ef46044d3a113266f1ebba9e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } }
I20260806 02:46:16.524854 10063 ts_tablet_manager.cc:1397] T 1d460a7404b54944ae05e39d92f94273 P 7d735936b19c4d7aa4b8db79c7694f1d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 02:46:16.525141 10064 ts_tablet_manager.cc:1428] T 1d460a7404b54944ae05e39d92f94273 P 1fa6b44ef46044d3a113266f1ebba9e4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 02:46:16.525592 10063 raft_consensus.cc:348] T 1d460a7404b54944ae05e39d92f94273 P 7d735936b19c4d7aa4b8db79c7694f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } }
I20260806 02:46:16.525866 10063 raft_consensus.cc:374] T 1d460a7404b54944ae05e39d92f94273 P 7d735936b19c4d7aa4b8db79c7694f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:16.525974 10063 raft_consensus.cc:729] T 1d460a7404b54944ae05e39d92f94273 P 7d735936b19c4d7aa4b8db79c7694f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d735936b19c4d7aa4b8db79c7694f1d, State: Initialized, Role: FOLLOWER
I20260806 02:46:16.526217 10063 consensus_queue.cc:260] T 1d460a7404b54944ae05e39d92f94273 P 7d735936b19c4d7aa4b8db79c7694f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } }
I20260806 02:46:16.526764 10063 ts_tablet_manager.cc:1428] T 1d460a7404b54944ae05e39d92f94273 P 7d735936b19c4d7aa4b8db79c7694f1d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260806 02:46:16.765069 10068 raft_consensus.cc:482] T 1d460a7404b54944ae05e39d92f94273 P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:46:16.765337 10068 raft_consensus.cc:504] T 1d460a7404b54944ae05e39d92f94273 P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } }
I20260806 02:46:16.766009 10068 leader_election.cc:288] T 1d460a7404b54944ae05e39d92f94273 P 89153066de564aab9ff1c28037bb286c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d735936b19c4d7aa4b8db79c7694f1d (127.1.210.65:33721), 1fa6b44ef46044d3a113266f1ebba9e4 (127.1.210.67:42521)
I20260806 02:46:16.766530 9704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d460a7404b54944ae05e39d92f94273" candidate_uuid: "89153066de564aab9ff1c28037bb286c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" is_pre_election: true
I20260806 02:46:16.766701 9987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d460a7404b54944ae05e39d92f94273" candidate_uuid: "89153066de564aab9ff1c28037bb286c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" is_pre_election: true
I20260806 02:46:16.766812 9704 raft_consensus.cc:2454] T 1d460a7404b54944ae05e39d92f94273 P 7d735936b19c4d7aa4b8db79c7694f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89153066de564aab9ff1c28037bb286c in term 0.
I20260806 02:46:16.766961 9987 raft_consensus.cc:2454] T 1d460a7404b54944ae05e39d92f94273 P 1fa6b44ef46044d3a113266f1ebba9e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89153066de564aab9ff1c28037bb286c in term 0.
I20260806 02:46:16.767351 9775 leader_election.cc:302] T 1d460a7404b54944ae05e39d92f94273 P 89153066de564aab9ff1c28037bb286c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d735936b19c4d7aa4b8db79c7694f1d, 89153066de564aab9ff1c28037bb286c; no voters:
I20260806 02:46:16.767652 10068 raft_consensus.cc:2781] T 1d460a7404b54944ae05e39d92f94273 P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:46:16.767763 10068 raft_consensus.cc:482] T 1d460a7404b54944ae05e39d92f94273 P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:46:16.767879 10068 raft_consensus.cc:3037] T 1d460a7404b54944ae05e39d92f94273 P 89153066de564aab9ff1c28037bb286c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:16.769974 10068 raft_consensus.cc:504] T 1d460a7404b54944ae05e39d92f94273 P 89153066de564aab9ff1c28037bb286c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } }
I20260806 02:46:16.770612 10068 leader_election.cc:288] T 1d460a7404b54944ae05e39d92f94273 P 89153066de564aab9ff1c28037bb286c [CANDIDATE]: Term 1 election: Requested vote from peers 7d735936b19c4d7aa4b8db79c7694f1d (127.1.210.65:33721), 1fa6b44ef46044d3a113266f1ebba9e4 (127.1.210.67:42521)
I20260806 02:46:16.771118 9704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d460a7404b54944ae05e39d92f94273" candidate_uuid: "89153066de564aab9ff1c28037bb286c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d735936b19c4d7aa4b8db79c7694f1d"
I20260806 02:46:16.771350 9704 raft_consensus.cc:3037] T 1d460a7404b54944ae05e39d92f94273 P 7d735936b19c4d7aa4b8db79c7694f1d [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:16.771304 9987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d460a7404b54944ae05e39d92f94273" candidate_uuid: "89153066de564aab9ff1c28037bb286c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fa6b44ef46044d3a113266f1ebba9e4"
I20260806 02:46:16.771553 9987 raft_consensus.cc:3037] T 1d460a7404b54944ae05e39d92f94273 P 1fa6b44ef46044d3a113266f1ebba9e4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:16.773767 9987 raft_consensus.cc:2454] T 1d460a7404b54944ae05e39d92f94273 P 1fa6b44ef46044d3a113266f1ebba9e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89153066de564aab9ff1c28037bb286c in term 1.
I20260806 02:46:16.774317 9704 raft_consensus.cc:2454] T 1d460a7404b54944ae05e39d92f94273 P 7d735936b19c4d7aa4b8db79c7694f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89153066de564aab9ff1c28037bb286c in term 1.
I20260806 02:46:16.774334 9775 leader_election.cc:302] T 1d460a7404b54944ae05e39d92f94273 P 89153066de564aab9ff1c28037bb286c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fa6b44ef46044d3a113266f1ebba9e4, 89153066de564aab9ff1c28037bb286c; no voters:
I20260806 02:46:16.774663 10068 raft_consensus.cc:2781] T 1d460a7404b54944ae05e39d92f94273 P 89153066de564aab9ff1c28037bb286c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:46:16.774885 10068 raft_consensus.cc:686] T 1d460a7404b54944ae05e39d92f94273 P 89153066de564aab9ff1c28037bb286c [term 1 LEADER]: Becoming Leader. State: Replica: 89153066de564aab9ff1c28037bb286c, State: Running, Role: LEADER
I20260806 02:46:16.775178 10068 consensus_queue.cc:237] T 1d460a7404b54944ae05e39d92f94273 P 89153066de564aab9ff1c28037bb286c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } }
I20260806 02:46:16.778164 9542 catalog_manager.cc:5495] T 1d460a7404b54944ae05e39d92f94273 P 89153066de564aab9ff1c28037bb286c reported cstate change: term changed from 0 to 1, leader changed from <none> to 89153066de564aab9ff1c28037bb286c (127.1.210.66). New cstate: current_term: 1 leader_uuid: "89153066de564aab9ff1c28037bb286c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89153066de564aab9ff1c28037bb286c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 } health_report { overall_health: UNKNOWN } } }
I20260806 02:46:16.832211 9987 raft_consensus.cc:1261] T 1d460a7404b54944ae05e39d92f94273 P 1fa6b44ef46044d3a113266f1ebba9e4 [term 1 FOLLOWER]: Refusing update from remote peer 89153066de564aab9ff1c28037bb286c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:46:16.832346 9704 raft_consensus.cc:1261] T 1d460a7404b54944ae05e39d92f94273 P 7d735936b19c4d7aa4b8db79c7694f1d [term 1 FOLLOWER]: Refusing update from remote peer 89153066de564aab9ff1c28037bb286c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:46:16.832798 10128 consensus_queue.cc:1035] T 1d460a7404b54944ae05e39d92f94273 P 89153066de564aab9ff1c28037bb286c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fa6b44ef46044d3a113266f1ebba9e4" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:46:16.833082 10068 consensus_queue.cc:1035] T 1d460a7404b54944ae05e39d92f94273 P 89153066de564aab9ff1c28037bb286c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d735936b19c4d7aa4b8db79c7694f1d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 33721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:46:16.881127 9542 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:46:16.881350 9542 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:46:16.884366 9542 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e49507f75394515a4d7645a0f7848c1: Sending DeleteTablet for 3 replicas of tablet 1d460a7404b54944ae05e39d92f94273
I20260806 02:46:16.885242 9682 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d460a7404b54944ae05e39d92f94273 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:46:16 UTC) from {username='oryx', principal='oryx/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:38874
I20260806 02:46:16.885288 9824 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d460a7404b54944ae05e39d92f94273 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:46:16 UTC) from {username='oryx', principal='oryx/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:53318
I20260806 02:46:16.885418 9966 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d460a7404b54944ae05e39d92f94273 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:46:16 UTC) from {username='oryx', principal='oryx/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:55050
I20260806 02:46:16.885573 10129 tablet_replica.cc:330] T 1d460a7404b54944ae05e39d92f94273 P 89153066de564aab9ff1c28037bb286c: stopping tablet replica
I20260806 02:46:16.885604 10131 tablet_replica.cc:330] T 1d460a7404b54944ae05e39d92f94273 P 7d735936b19c4d7aa4b8db79c7694f1d: stopping tablet replica
I20260806 02:46:16.885756 10130 tablet_replica.cc:330] T 1d460a7404b54944ae05e39d92f94273 P 1fa6b44ef46044d3a113266f1ebba9e4: stopping tablet replica
I20260806 02:46:16.885771 10131 raft_consensus.cc:2229] T 1d460a7404b54944ae05e39d92f94273 P 7d735936b19c4d7aa4b8db79c7694f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:46:16.885771 10129 raft_consensus.cc:2229] T 1d460a7404b54944ae05e39d92f94273 P 89153066de564aab9ff1c28037bb286c [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:46:16.885938 10130 raft_consensus.cc:2229] T 1d460a7404b54944ae05e39d92f94273 P 1fa6b44ef46044d3a113266f1ebba9e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:46:16.885975 10131 raft_consensus.cc:2258] T 1d460a7404b54944ae05e39d92f94273 P 7d735936b19c4d7aa4b8db79c7694f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:46:16.886161 10130 raft_consensus.cc:2258] T 1d460a7404b54944ae05e39d92f94273 P 1fa6b44ef46044d3a113266f1ebba9e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:46:16.886200 10129 raft_consensus.cc:2258] T 1d460a7404b54944ae05e39d92f94273 P 89153066de564aab9ff1c28037bb286c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:46:16.886608 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 9617
I20260806 02:46:16.887169 10131 ts_tablet_manager.cc:1905] T 1d460a7404b54944ae05e39d92f94273 P 7d735936b19c4d7aa4b8db79c7694f1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:46:16.887652 10130 ts_tablet_manager.cc:1905] T 1d460a7404b54944ae05e39d92f94273 P 1fa6b44ef46044d3a113266f1ebba9e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:46:16.890377 10129 ts_tablet_manager.cc:1905] T 1d460a7404b54944ae05e39d92f94273 P 89153066de564aab9ff1c28037bb286c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:46:16.894438 10130 ts_tablet_manager.cc:1918] T 1d460a7404b54944ae05e39d92f94273 P 1fa6b44ef46044d3a113266f1ebba9e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:46:16.894594 10130 log.cc:1196] T 1d460a7404b54944ae05e39d92f94273 P 1fa6b44ef46044d3a113266f1ebba9e4: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-2/wal/wals/1d460a7404b54944ae05e39d92f94273
I20260806 02:46:16.895138 10130 ts_tablet_manager.cc:1939] T 1d460a7404b54944ae05e39d92f94273 P 1fa6b44ef46044d3a113266f1ebba9e4: Deleting consensus metadata
I20260806 02:46:16.896485 9528 catalog_manager.cc:4841] TS 1fa6b44ef46044d3a113266f1ebba9e4 (127.1.210.67:42521): tablet 1d460a7404b54944ae05e39d92f94273 (table test-table [id=601bd31414c9401aaa01c8259a6c9449]) successfully deleted
I20260806 02:46:16.897886 10131 ts_tablet_manager.cc:1918] T 1d460a7404b54944ae05e39d92f94273 P 7d735936b19c4d7aa4b8db79c7694f1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:46:16.898069 10131 log.cc:1196] T 1d460a7404b54944ae05e39d92f94273 P 7d735936b19c4d7aa4b8db79c7694f1d: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-0/wal/wals/1d460a7404b54944ae05e39d92f94273
I20260806 02:46:16.898488 10131 ts_tablet_manager.cc:1939] T 1d460a7404b54944ae05e39d92f94273 P 7d735936b19c4d7aa4b8db79c7694f1d: Deleting consensus metadata
I20260806 02:46:16.898903 10129 ts_tablet_manager.cc:1918] T 1d460a7404b54944ae05e39d92f94273 P 89153066de564aab9ff1c28037bb286c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:46:16.899086 10129 log.cc:1196] T 1d460a7404b54944ae05e39d92f94273 P 89153066de564aab9ff1c28037bb286c: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984304136196-1865-0/minicluster-data/ts-1/wal/wals/1d460a7404b54944ae05e39d92f94273
I20260806 02:46:16.899516 10129 ts_tablet_manager.cc:1939] T 1d460a7404b54944ae05e39d92f94273 P 89153066de564aab9ff1c28037bb286c: Deleting consensus metadata
I20260806 02:46:16.899744 9528 catalog_manager.cc:4841] TS 7d735936b19c4d7aa4b8db79c7694f1d (127.1.210.65:33721): tablet 1d460a7404b54944ae05e39d92f94273 (table test-table [id=601bd31414c9401aaa01c8259a6c9449]) successfully deleted
I20260806 02:46:16.900627 9530 catalog_manager.cc:4841] TS 89153066de564aab9ff1c28037bb286c (127.1.210.66:37491): tablet 1d460a7404b54944ae05e39d92f94273 (table test-table [id=601bd31414c9401aaa01c8259a6c9449]) successfully deleted
Aug 06 02:46:16 dist-test-slave-8tdl krb5kdc[9499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.210.65@KRBTEST.COM
I20260806 02:46:16.912624 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:16.889072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38888 (local address 127.1.210.65:33721)
0806 02:46:16.896651 (+ 7579us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:16.896658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:16.896682 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:46:16.896751 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:16.896756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:16.896852 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:16.896999 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:16.898835 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:16.899896 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:16.901369 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:16.901783 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:16.901935 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:16.906368 (+ 4433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:16.906388 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:16.906407 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:16.906445 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:16.909484 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:16.910254 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:16.910262 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:16.910264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:16.910408 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:16.910858 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:16.910863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:16.910867 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:16.911273 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:16.911626 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:16.911956 (+ 330us) server_negotiation.cc:299] Negotiation successful
0806 02:46:16.912220 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7419,"thread_start_us":7638,"threads_started":1}
I20260806 02:46:16.913295 9750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:46:17.069494 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 9617
I20260806 02:46:17.093993 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 9759
Aug 06 02:46:17 dist-test-slave-8tdl krb5kdc[9499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.210.66@KRBTEST.COM
I20260806 02:46:17.113703 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:17.095288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53350 (local address 127.1.210.66:37491)
0806 02:46:17.095804 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:17.095809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:17.095826 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:46:17.095885 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:17.095888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:17.095965 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:17.096065 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:17.097682 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:17.098438 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:17.100434 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:17.100988 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:17.101269 (+ 281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:17.107051 (+ 5782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:17.107071 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:17.107095 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:17.107148 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:17.110116 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:17.111252 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:17.111260 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:17.111263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:17.111437 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:17.111887 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:17.111891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:17.111894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:17.112383 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:17.112663 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:17.113077 (+ 414us) server_negotiation.cc:299] Negotiation successful
0806 02:46:17.113292 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":309,"threads_started":1}
I20260806 02:46:17.114578 9890 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:46:17.247308 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 9759
I20260806 02:46:17.271780 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 9900
Aug 06 02:46:17 dist-test-slave-8tdl krb5kdc[9499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.210.67@KRBTEST.COM
I20260806 02:46:17.288233 10167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:17.272651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55066 (local address 127.1.210.67:42521)
0806 02:46:17.273193 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:17.273198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:17.273214 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:46:17.273285 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:17.273288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:17.273350 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:17.273497 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:17.274923 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:17.275652 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:17.276977 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:17.277371 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:17.277542 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:17.282360 (+ 4818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:17.282378 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:17.282396 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:17.282437 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:17.285441 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:17.286220 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:17.286227 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:17.286230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:17.286371 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:17.286836 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:17.286840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:17.286842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:17.287253 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:17.287488 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:17.287774 (+ 286us) server_negotiation.cc:299] Negotiation successful
0806 02:46:17.287960 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":269,"threads_started":1}
I20260806 02:46:17.289098 10032 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:46:17.389711 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 9900
I20260806 02:46:17.407925 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 9512
I20260806 02:46:17.409212 9585 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:46:17.533921 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 9512
2026-08-06T02:46:17Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7393 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:46:17 dist-test-slave-8tdl krb5kdc[10172](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:46:17 dist-test-slave-8tdl krb5kdc[10172](info): set up 2 sockets
Aug 06 02:46:17 dist-test-slave-8tdl krb5kdc[10172](info): commencing operation
krb5kdc: starting...
W20260806 02:46:18.699043 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:46:18 dist-test-slave-8tdl krb5kdc[10172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:46:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:46:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
I20260806 02:46:18.970992 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:41831
--webserver_interface=127.1.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:36297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:41831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:46:19.312948 10185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:19.313256 10185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:19.313409 10185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:19.321169 10185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:46:19.321301 10185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:19.321393 10185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:19.321499 10185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:46:19.321589 10185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:46:19.332438 10185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:36297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:41831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:41831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:46:19.333869 10185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:19.335353 10185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:19.348527 10190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:19.348841 10191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:19.348896 10193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:19.349429 10185 server_base.cc:1034] running on GCE node
I20260806 02:46:19.350445 10185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:19.352084 10185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:19.353374 10185 hybrid_clock.cc:648] HybridClock initialized: now 1785984379353318 us; error 40 us; skew 500 ppm
I20260806 02:46:19.353799 10185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-06T02:46:19Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1886 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:46:19 dist-test-slave-8tdl krb5kdc[10199](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:46:19 dist-test-slave-8tdl krb5kdc[10199](info): set up 2 sockets
Aug 06 02:46:19 dist-test-slave-8tdl krb5kdc[10199](info): commencing operation
krb5kdc: starting...
W20260806 02:46:20.578970 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.003s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:46:20 dist-test-slave-8tdl krb5kdc[10199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:46:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:46:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
I20260806 02:46:20.847417 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:33657
--webserver_interface=127.1.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:39019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:33657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:46:21.203976 10212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:21.204370 10212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:21.204552 10212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:21.212221 10212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:46:21.212396 10212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:21.212509 10212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:21.212618 10212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:46:21.212711 10212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:46:21.223138 10212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:39019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:33657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:33657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:46:21.224669 10212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:21.226056 10212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:21.240295 10218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:21.240244 10219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:21.240193 10221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:21.241825 10212 server_base.cc:1034] running on GCE node
I20260806 02:46:21.242849 10212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:21.244620 10212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:21.246043 10212 hybrid_clock.cc:648] HybridClock initialized: now 1785984381245907 us; error 115 us; skew 500 ppm
I20260806 02:46:21.246472 10212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:46:21 dist-test-slave-8tdl krb5kdc[10199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984381, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:21.255668 10212 init.cc:377] Logged in from keytab as kudu/127.1.210.126@KRBTEST.COM (short username kudu)
I20260806 02:46:21.257869 10212 webserver.cc:460] Webserver started at http://127.1.210.126:41051/ using document root <none> and password file <none>
I20260806 02:46:21.258837 10212 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:21.259014 10212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:21.259374 10212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:21.262806 10212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "59bc1d3c57e44bc8ae33167f93d0a72e"
format_stamp: "Formatted at 2026-08-06 02:46:21 on dist-test-slave-8tdl"
server_key: "653af8323ef5725674d376b30f5da24f"
server_key_iv: "639187d92cc5b46e06b0a0cd3d2be837"
server_key_version: "encryptionkey@0"
I20260806 02:46:21.263720 10212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "59bc1d3c57e44bc8ae33167f93d0a72e"
format_stamp: "Formatted at 2026-08-06 02:46:21 on dist-test-slave-8tdl"
server_key: "653af8323ef5725674d376b30f5da24f"
server_key_iv: "639187d92cc5b46e06b0a0cd3d2be837"
server_key_version: "encryptionkey@0"
I20260806 02:46:21.269873 10212 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260806 02:46:21.274334 10228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:21.276113 10212 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:46:21.276377 10212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "59bc1d3c57e44bc8ae33167f93d0a72e"
format_stamp: "Formatted at 2026-08-06 02:46:21 on dist-test-slave-8tdl"
server_key: "653af8323ef5725674d376b30f5da24f"
server_key_iv: "639187d92cc5b46e06b0a0cd3d2be837"
server_key_version: "encryptionkey@0"
I20260806 02:46:21.276641 10212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:21.299094 10212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:21.305006 10212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:21.305439 10212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:21.327600 10212 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:33657
I20260806 02:46:21.327636 10279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:33657 every 8 connection(s)
I20260806 02:46:21.329180 10212 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:46:21.333609 10280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:21.335577 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 10212
I20260806 02:46:21.335825 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:46:21.336534 1865 external_mini_cluster.cc:1428] Setting key 4f10d21814df587c5ef95c9925778865
I20260806 02:46:21.350376 10280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e: Bootstrap starting.
Aug 06 02:46:21 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:46:21.354796 10280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:21.356400 10280 log.cc:824] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e: Log is configured to *not* fsync() on all Append() calls
I20260806 02:46:21.360463 10280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e: No bootstrap required, opened a new log
I20260806 02:46:21.365398 10283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:21.338839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49680 (local address 127.1.210.126:33657)
0806 02:46:21.339760 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:21.339774 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:21.339819 (+ 45us) server_negotiation.cc:407] Connection header received
0806 02:46:21.341556 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:21.341586 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:21.342038 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:21.342427 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:21.344716 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:21.346383 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:21.348744 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:21.349571 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:21.349736 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:21.355220 (+ 5484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:21.355236 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:21.355265 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:21.355333 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:21.359729 (+ 4396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:21.360656 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:21.360667 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:21.360677 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:21.360923 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:21.361411 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:21.361418 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:21.361421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:21.362243 (+ 822us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:21.362619 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:21.363179 (+ 560us) server_negotiation.cc:299] Negotiation successful
0806 02:46:21.363737 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":420,"threads_started":1}
I20260806 02:46:21.371248 10280 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59bc1d3c57e44bc8ae33167f93d0a72e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 33657 } }
I20260806 02:46:21.371706 10280 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:21.371806 10280 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59bc1d3c57e44bc8ae33167f93d0a72e, State: Initialized, Role: FOLLOWER
I20260806 02:46:21.372419 10280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59bc1d3c57e44bc8ae33167f93d0a72e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 33657 } }
I20260806 02:46:21.372644 10280 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:46:21.372783 10280 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:46:21.372918 10280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:21.375726 10280 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59bc1d3c57e44bc8ae33167f93d0a72e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 33657 } }
I20260806 02:46:21.376250 10280 leader_election.cc:302] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59bc1d3c57e44bc8ae33167f93d0a72e; no voters:
I20260806 02:46:21.376870 10280 leader_election.cc:288] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:46:21.377147 10285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:46:21.377919 10285 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e [term 1 LEADER]: Becoming Leader. State: Replica: 59bc1d3c57e44bc8ae33167f93d0a72e, State: Running, Role: LEADER
I20260806 02:46:21.378525 10285 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59bc1d3c57e44bc8ae33167f93d0a72e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 33657 } }
I20260806 02:46:21.379384 10280 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:46:21.381994 10286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59bc1d3c57e44bc8ae33167f93d0a72e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59bc1d3c57e44bc8ae33167f93d0a72e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 33657 } } }
I20260806 02:46:21.382503 10286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e [sys.catalog]: This master's current role is: LEADER
I20260806 02:46:21.382067 10287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59bc1d3c57e44bc8ae33167f93d0a72e. Latest consensus state: current_term: 1 leader_uuid: "59bc1d3c57e44bc8ae33167f93d0a72e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59bc1d3c57e44bc8ae33167f93d0a72e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 33657 } } }
I20260806 02:46:21.382740 10287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e [sys.catalog]: This master's current role is: LEADER
I20260806 02:46:21.391304 10291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:46:21.396054 10291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:46:21.405702 10291 catalog_manager.cc:1347] Generated new cluster ID: c97f78315a63419aa217100a6b728e67
I20260806 02:46:21.405854 10291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:46:21.421975 10291 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:46:21.424041 10291 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:46:21.431866 10291 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e: Generated new TSK 0
I20260806 02:46:21.433102 10291 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab.
Entry for principal kudu/127.1.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab.
I20260806 02:46:21.507519 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.65:0
--local_ip_for_outbound_sockets=127.1.210.65
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:33657
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:39019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:46:21.884138 10306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:21.884438 10306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:21.884598 10306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:21.892637 10306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:21.892773 10306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:21.892948 10306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.65
I20260806 02:46:21.903602 10306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:39019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:33657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.65
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:46:21.905110 10306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:21.906558 10306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:21.921573 10312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:21.921757 10315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:21.921797 10313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:21.923040 10306 server_base.cc:1034] running on GCE node
I20260806 02:46:21.923799 10306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:21.925114 10306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:21.926373 10306 hybrid_clock.cc:648] HybridClock initialized: now 1785984381926331 us; error 40 us; skew 500 ppm
I20260806 02:46:21.926853 10306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:46:21 dist-test-slave-8tdl krb5kdc[10199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984381, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:21.930655 10306 init.cc:377] Logged in from keytab as kudu/127.1.210.65@KRBTEST.COM (short username kudu)
I20260806 02:46:21.932608 10306 webserver.cc:460] Webserver started at http://127.1.210.65:34559/ using document root <none> and password file <none>
I20260806 02:46:21.933377 10306 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:21.933493 10306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:21.933784 10306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:21.937314 10306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-0/data/instance:
uuid: "a0ec03f9f6254c838b83d6a4fdafe0b9"
format_stamp: "Formatted at 2026-08-06 02:46:21 on dist-test-slave-8tdl"
server_key: "b9966b9518db00634ff3ac3fabb4a05d"
server_key_iv: "2d059a89e3693b98b15e0a2b89484f83"
server_key_version: "encryptionkey@0"
I20260806 02:46:21.938412 10306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance:
uuid: "a0ec03f9f6254c838b83d6a4fdafe0b9"
format_stamp: "Formatted at 2026-08-06 02:46:21 on dist-test-slave-8tdl"
server_key: "b9966b9518db00634ff3ac3fabb4a05d"
server_key_iv: "2d059a89e3693b98b15e0a2b89484f83"
server_key_version: "encryptionkey@0"
I20260806 02:46:21.944516 10306 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260806 02:46:21.948839 10322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:21.950469 10306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260806 02:46:21.950681 10306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-0/wal
uuid: "a0ec03f9f6254c838b83d6a4fdafe0b9"
format_stamp: "Formatted at 2026-08-06 02:46:21 on dist-test-slave-8tdl"
server_key: "b9966b9518db00634ff3ac3fabb4a05d"
server_key_iv: "2d059a89e3693b98b15e0a2b89484f83"
server_key_version: "encryptionkey@0"
I20260806 02:46:21.950963 10306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:21.979841 10306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:21.985421 10306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:21.985786 10306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:21.987236 10306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:46:21.989645 10306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:21.989774 10306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:21.989923 10306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:21.989988 10306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:22.030978 10306 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.65:37937
I20260806 02:46:22.031136 10434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.65:37937 every 8 connection(s)
I20260806 02:46:22.032418 10306 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
I20260806 02:46:22.037497 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 10306
I20260806 02:46:22.037727 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance
I20260806 02:46:22.038537 1865 external_mini_cluster.cc:1428] Setting key 93bc41bf32f12a4965d98615819e8a77
Aug 06 02:46:22 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984381, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:46:22.055645 10437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:22.035463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:48061 (local address 127.1.210.126:33657)
0806 02:46:22.035990 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:22.035998 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:22.037063 (+ 1065us) server_negotiation.cc:407] Connection header received
0806 02:46:22.039298 (+ 2235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:22.039304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:22.039407 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:22.039532 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:22.041369 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:22.042126 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:22.044117 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:22.044558 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:22.044710 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:22.049164 (+ 4454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:22.049171 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:22.049175 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:22.049214 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:22.052105 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:22.052999 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:22.053013 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:22.053015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:22.053148 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:22.053645 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:22.053654 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:22.053656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:22.053952 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:22.054126 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:22.055006 (+ 880us) server_negotiation.cc:299] Negotiation successful
0806 02:46:22.055252 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":261,"threads_started":1}
I20260806 02:46:22.056792 10438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:22.036101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.65:48061)
0806 02:46:22.036851 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:22.036924 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:22.038990 (+ 2066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:22.039815 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:22.039825 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:22.040477 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:22.041172 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:22.041194 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:22.042300 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:22.042305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:22.043868 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:22.043889 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:22.044839 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:22.044846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:22.045118 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:22.045940 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:22.045965 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:22.048851 (+ 2886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:22.052309 (+ 3458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:22.052319 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:22.052340 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:22.052845 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:22.053269 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:22.053273 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:22.053277 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:22.053504 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:22.054101 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:22.054110 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:22.054505 (+ 395us) client_negotiation.cc:769] Sending connection context
0806 02:46:22.054848 (+ 343us) client_negotiation.cc:240] Negotiation successful
0806 02:46:22.055492 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":281,"threads_started":1}
I20260806 02:46:22.058657 10435 heartbeater.cc:347] Connected to a master server at 127.1.210.126:33657
I20260806 02:46:22.059067 10435 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:22.059870 10435 heartbeater.cc:511] Master 127.1.210.126:33657 requested a full tablet report, sending...
I20260806 02:46:22.062204 10245 ts_manager.cc:194] Registered new tserver with Master: a0ec03f9f6254c838b83d6a4fdafe0b9 (127.1.210.65:37937)
WARNING: no policy specified for kudu/127.1.210.66@KRBTEST.COM; defaulting to no policy
I20260806 02:46:22.063896 10245 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.65@KRBTEST.COM'} at 127.1.210.65:48061
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab.
Entry for principal kudu/127.1.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab.
I20260806 02:46:22.098652 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.66:0
--local_ip_for_outbound_sockets=127.1.210.66
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:33657
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:39019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:46:22.456931 10441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:22.457237 10441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:22.457393 10441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:22.465224 10441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:22.465346 10441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:22.465468 10441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.66
I20260806 02:46:22.475745 10441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:39019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:33657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.66
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:46:22.477092 10441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:22.478529 10441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:22.493149 10450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:22.493212 10447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:22.493191 10448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:22.493705 10441 server_base.cc:1034] running on GCE node
I20260806 02:46:22.494805 10441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:22.495987 10441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:22.497265 10441 hybrid_clock.cc:648] HybridClock initialized: now 1785984382497211 us; error 50 us; skew 500 ppm
I20260806 02:46:22.497659 10441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:46:22 dist-test-slave-8tdl krb5kdc[10199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984382, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:22.501106 10441 init.cc:377] Logged in from keytab as kudu/127.1.210.66@KRBTEST.COM (short username kudu)
I20260806 02:46:22.502818 10441 webserver.cc:460] Webserver started at http://127.1.210.66:42807/ using document root <none> and password file <none>
I20260806 02:46:22.503572 10441 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:22.503675 10441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:22.503944 10441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:22.507172 10441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-1/data/instance:
uuid: "cabcbf1a307741109d266cec6b5f40f6"
format_stamp: "Formatted at 2026-08-06 02:46:22 on dist-test-slave-8tdl"
server_key: "ac5e9d84f4d53628477d656e2d1babd6"
server_key_iv: "0f5800facc73943f6e5e575d0974b87c"
server_key_version: "encryptionkey@0"
I20260806 02:46:22.508039 10441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance:
uuid: "cabcbf1a307741109d266cec6b5f40f6"
format_stamp: "Formatted at 2026-08-06 02:46:22 on dist-test-slave-8tdl"
server_key: "ac5e9d84f4d53628477d656e2d1babd6"
server_key_iv: "0f5800facc73943f6e5e575d0974b87c"
server_key_version: "encryptionkey@0"
I20260806 02:46:22.513759 10441 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260806 02:46:22.518404 10457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:22.520246 10441 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260806 02:46:22.520490 10441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-1/wal
uuid: "cabcbf1a307741109d266cec6b5f40f6"
format_stamp: "Formatted at 2026-08-06 02:46:22 on dist-test-slave-8tdl"
server_key: "ac5e9d84f4d53628477d656e2d1babd6"
server_key_iv: "0f5800facc73943f6e5e575d0974b87c"
server_key_version: "encryptionkey@0"
I20260806 02:46:22.520695 10441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:22.536252 10441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:22.541059 10441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:22.541397 10441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:22.542491 10441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:46:22.544018 10441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:22.544104 10441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:22.544278 10441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:22.544345 10441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:22.579362 10441 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.66:44759
I20260806 02:46:22.579382 10569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.66:44759 every 8 connection(s)
I20260806 02:46:22.580983 10441 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
I20260806 02:46:22.586889 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 10441
I20260806 02:46:22.587075 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance
I20260806 02:46:22.587639 1865 external_mini_cluster.cc:1428] Setting key 8674b7aedeff1c026d574f44073181fc
Aug 06 02:46:22 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984382, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:46:22.607473 10572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:22.584085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:45881 (local address 127.1.210.126:33657)
0806 02:46:22.584586 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:22.584590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:22.585747 (+ 1157us) server_negotiation.cc:407] Connection header received
0806 02:46:22.588469 (+ 2722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:22.588476 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:22.588611 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:22.588741 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:22.590794 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:22.591542 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:22.594194 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:22.594660 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:22.594834 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:22.599812 (+ 4978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:22.599818 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:22.599822 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:22.599860 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:22.603513 (+ 3653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:22.604445 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:22.604454 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:22.604460 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:22.604641 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:22.605117 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:22.605125 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:22.605131 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:22.605589 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:22.605752 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:22.606605 (+ 853us) server_negotiation.cc:299] Negotiation successful
0806 02:46:22.606936 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":258,"threads_started":1}
I20260806 02:46:22.608258 10573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:22.584757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.66:45881)
0806 02:46:22.585560 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:22.585608 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:22.588081 (+ 2473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:22.589018 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:22.589027 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:22.589706 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:22.590563 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:22.590593 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:22.591720 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:22.591729 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:22.593995 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:22.594016 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:22.594982 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:22.594992 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:22.595352 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:22.596163 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:22.596200 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:22.599503 (+ 3303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:22.603718 (+ 4215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:22.603727 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:22.603743 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:22.604274 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:22.604768 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:22.604772 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:22.604775 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:22.604983 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:22.605748 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:22.605756 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:22.606159 (+ 403us) client_negotiation.cc:769] Sending connection context
0806 02:46:22.606441 (+ 282us) client_negotiation.cc:240] Negotiation successful
0806 02:46:22.606988 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":324,"threads_started":1}
I20260806 02:46:22.610080 10570 heartbeater.cc:347] Connected to a master server at 127.1.210.126:33657
I20260806 02:46:22.610442 10570 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:22.611290 10570 heartbeater.cc:511] Master 127.1.210.126:33657 requested a full tablet report, sending...
I20260806 02:46:22.613129 10245 ts_manager.cc:194] Registered new tserver with Master: cabcbf1a307741109d266cec6b5f40f6 (127.1.210.66:44759)
I20260806 02:46:22.614480 10245 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.66@KRBTEST.COM'} at 127.1.210.66:45881
WARNING: no policy specified for kudu/127.1.210.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab.
Entry for principal kudu/127.1.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab.
I20260806 02:46:22.651320 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.67:0
--local_ip_for_outbound_sockets=127.1.210.67
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:33657
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:39019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:46:23.006165 10577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:23.006523 10577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:23.006685 10577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:23.014487 10577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:23.014631 10577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:23.014842 10577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.67
I20260806 02:46:23.025588 10577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:39019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:33657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.67
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:46:23.027005 10577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:23.028851 10577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:23.044162 10582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:23.044195 10583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:23.045331 10585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:23.045305 10577 server_base.cc:1034] running on GCE node
I20260806 02:46:23.046099 10577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:23.047223 10577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:23.048475 10577 hybrid_clock.cc:648] HybridClock initialized: now 1785984383048421 us; error 32 us; skew 500 ppm
I20260806 02:46:23.048874 10577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:46:23 dist-test-slave-8tdl krb5kdc[10199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984383, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:23.053876 10577 init.cc:377] Logged in from keytab as kudu/127.1.210.67@KRBTEST.COM (short username kudu)
I20260806 02:46:23.055823 10577 webserver.cc:460] Webserver started at http://127.1.210.67:44065/ using document root <none> and password file <none>
I20260806 02:46:23.056627 10577 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:23.056780 10577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:23.057116 10577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:23.060498 10577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-2/data/instance:
uuid: "6e112115b2774648be5066e86a590308"
format_stamp: "Formatted at 2026-08-06 02:46:23 on dist-test-slave-8tdl"
server_key: "52e44744684f0cfc38c13d08d9862bd2"
server_key_iv: "ad9298d1a25a2e7a8e47772bd7d9c5c0"
server_key_version: "encryptionkey@0"
I20260806 02:46:23.061367 10577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance:
uuid: "6e112115b2774648be5066e86a590308"
format_stamp: "Formatted at 2026-08-06 02:46:23 on dist-test-slave-8tdl"
server_key: "52e44744684f0cfc38c13d08d9862bd2"
server_key_iv: "ad9298d1a25a2e7a8e47772bd7d9c5c0"
server_key_version: "encryptionkey@0"
I20260806 02:46:23.067494 10577 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260806 02:46:23.067824 10435 heartbeater.cc:503] Master 127.1.210.126:33657 was elected leader, sending a full tablet report...
I20260806 02:46:23.072388 10592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:23.074043 10577 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:46:23.074268 10577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-2/wal
uuid: "6e112115b2774648be5066e86a590308"
format_stamp: "Formatted at 2026-08-06 02:46:23 on dist-test-slave-8tdl"
server_key: "52e44744684f0cfc38c13d08d9862bd2"
server_key_iv: "ad9298d1a25a2e7a8e47772bd7d9c5c0"
server_key_version: "encryptionkey@0"
I20260806 02:46:23.074508 10577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:23.100860 10577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:23.106269 10577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:23.106647 10577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:23.107980 10577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:46:23.109614 10577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:23.109699 10577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:23.109794 10577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:23.109844 10577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:23.147632 10577 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.67:34335
I20260806 02:46:23.147684 10705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.67:34335 every 8 connection(s)
I20260806 02:46:23.149021 10577 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
I20260806 02:46:23.154382 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 10577
I20260806 02:46:23.154621 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance
I20260806 02:46:23.155205 1865 external_mini_cluster.cc:1428] Setting key 78ce6d6e426526d612eb1722f3ac01f8
Aug 06 02:46:23 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984383, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:46:23.172084 10708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:23.152145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:47095 (local address 127.1.210.126:33657)
0806 02:46:23.152647 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:23.152652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:23.153677 (+ 1025us) server_negotiation.cc:407] Connection header received
0806 02:46:23.155911 (+ 2234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:23.155916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:23.156007 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:23.156101 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:23.158142 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.159089 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.160810 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.161246 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.161359 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:23.165244 (+ 3885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:23.165250 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:23.165254 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:23.165292 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:23.168218 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:23.169098 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:23.169104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:23.169106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:23.169285 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:23.169888 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:23.169893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:23.169896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:23.170289 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:23.170475 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:23.171561 (+ 1086us) server_negotiation.cc:299] Negotiation successful
0806 02:46:23.171754 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":237,"threads_started":1}
I20260806 02:46:23.173051 10709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:23.152730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.67:47095)
0806 02:46:23.153493 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:23.153544 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:23.155580 (+ 2036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:23.156664 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:23.156675 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:23.157302 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:23.157957 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:23.157986 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.159248 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.159252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:23.160636 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:23.160650 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.161504 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.161510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:23.161740 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:23.162487 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:23.162511 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:23.164991 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:23.168383 (+ 3392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:23.168393 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:23.168413 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:23.168911 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:23.169440 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:23.169447 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:23.169451 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:23.169753 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:23.170461 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:23.170467 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:23.171070 (+ 603us) client_negotiation.cc:769] Sending connection context
0806 02:46:23.171413 (+ 343us) client_negotiation.cc:240] Negotiation successful
0806 02:46:23.171849 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":312,"threads_started":1}
I20260806 02:46:23.174613 10706 heartbeater.cc:347] Connected to a master server at 127.1.210.126:33657
I20260806 02:46:23.175066 10706 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:23.175876 10706 heartbeater.cc:511] Master 127.1.210.126:33657 requested a full tablet report, sending...
I20260806 02:46:23.177609 10245 ts_manager.cc:194] Registered new tserver with Master: 6e112115b2774648be5066e86a590308 (127.1.210.67:34335)
I20260806 02:46:23.178526 10245 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.67@KRBTEST.COM'} at 127.1.210.67:47095
I20260806 02:46:23.184309 1865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.65.keytab.
Entry for principal oryx/127.1.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.65.keytab.
WARNING: no policy specified for oryx/127.1.210.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.210.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.66.keytab.
Entry for principal oryx/127.1.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.66.keytab.
WARNING: no policy specified for oryx/127.1.210.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.210.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.67.keytab.
Entry for principal oryx/127.1.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.67.keytab.
I20260806 02:46:23.358618 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 10306
Aug 06 02:46:23 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.65@KRBTEST.COM
I20260806 02:46:23.377059 10717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:23.359781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56648 (local address 127.1.210.65:37937)
0806 02:46:23.360549 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:23.360557 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:23.360585 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:46:23.360656 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:23.360662 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:23.360899 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:23.361065 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:23.362610 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.363770 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.365230 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.365900 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.366037 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:23.370511 (+ 4474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:23.370528 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:23.370545 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:23.370586 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:23.374194 (+ 3608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:23.375076 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:23.375084 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:23.375086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:23.375226 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:23.375571 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:23.375575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:23.375577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:23.375925 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:23.376200 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:23.376505 (+ 305us) server_negotiation.cc:299] Negotiation successful
0806 02:46:23.376720 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":418,"threads_started":1}
I20260806 02:46:23.378497 10430 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:46:23.463629 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 10306
I20260806 02:46:23.482028 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 10441
Aug 06 02:46:23 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.66@KRBTEST.COM
I20260806 02:46:23.499727 10719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:23.483077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39268 (local address 127.1.210.66:44759)
0806 02:46:23.483670 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:23.483677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:23.483705 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:46:23.483767 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:23.483773 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:23.483992 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:23.484128 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:23.485413 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.486451 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.488097 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.488727 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.488909 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:23.493349 (+ 4440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:23.493363 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:23.493378 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:23.493414 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:23.496593 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:23.497397 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:23.497405 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:23.497407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:23.497541 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:23.497972 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:23.497976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:23.497978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:23.498333 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:23.498685 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:23.499073 (+ 388us) server_negotiation.cc:299] Negotiation successful
0806 02:46:23.499358 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":292,"threads_started":1}
I20260806 02:46:23.501148 10565 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:46:23.590869 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 10441
I20260806 02:46:23.608038 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 10577
Aug 06 02:46:23 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.67@KRBTEST.COM
I20260806 02:46:23.626649 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:23.609116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38206 (local address 127.1.210.67:34335)
0806 02:46:23.609652 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:23.609660 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:23.609693 (+ 33us) server_negotiation.cc:407] Connection header received
0806 02:46:23.609758 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:23.609765 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:23.610001 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:23.610192 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:23.611823 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.612949 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.614272 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.615174 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.615303 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:23.620007 (+ 4704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:23.620023 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:23.620039 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:23.620095 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:23.623454 (+ 3359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:23.624272 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:23.624280 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:23.624283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:23.624417 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:23.624869 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:23.624873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:23.624875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:23.625259 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:23.625488 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:23.625981 (+ 493us) server_negotiation.cc:299] Negotiation successful
0806 02:46:23.626330 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":243,"threads_started":1}
I20260806 02:46:23.628041 10701 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:46:23.713805 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 10577
I20260806 02:46:23.733994 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 10212
I20260806 02:46:23.735450 10275 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:46:23.843982 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 10212
I20260806 02:46:23.865635 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:33657
--webserver_interface=127.1.210.126
--webserver_port=41051
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:39019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:33657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:46:24.218039 10724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:24.218372 10724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:24.218511 10724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:24.226152 10724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:46:24.226285 10724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:24.226336 10724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:24.226454 10724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:46:24.226547 10724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:46:24.237258 10724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:39019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:33657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:33657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=41051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:46:24.238797 10724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:24.240244 10724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:24.253132 10732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:24.253085 10730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:24.253348 10729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:24.253997 10724 server_base.cc:1034] running on GCE node
I20260806 02:46:24.254874 10724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:24.256480 10724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:24.257721 10724 hybrid_clock.cc:648] HybridClock initialized: now 1785984384257673 us; error 36 us; skew 500 ppm
I20260806 02:46:24.258138 10724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:46:24 dist-test-slave-8tdl krb5kdc[10199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984384, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:24.261596 10724 init.cc:377] Logged in from keytab as kudu/127.1.210.126@KRBTEST.COM (short username kudu)
I20260806 02:46:24.263346 10724 webserver.cc:460] Webserver started at http://127.1.210.126:41051/ using document root <none> and password file <none>
I20260806 02:46:24.264148 10724 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:24.264274 10724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:24.270824 10724 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 02:46:24.274296 10739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:24.276062 10724 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:46:24.276242 10724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "59bc1d3c57e44bc8ae33167f93d0a72e"
format_stamp: "Formatted at 2026-08-06 02:46:21 on dist-test-slave-8tdl"
server_key: "653af8323ef5725674d376b30f5da24f"
server_key_iv: "639187d92cc5b46e06b0a0cd3d2be837"
server_key_version: "encryptionkey@0"
I20260806 02:46:24.277357 10724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:24.294358 10724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:24.299335 10724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:24.299578 10724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:24.320590 10724 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:33657
I20260806 02:46:24.320619 10792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:33657 every 8 connection(s)
I20260806 02:46:24.322221 10724 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:46:24.323467 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 10724
I20260806 02:46:24.324620 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.65:37937
--local_ip_for_outbound_sockets=127.1.210.65
--tserver_master_addrs=127.1.210.126:33657
--webserver_port=34559
--webserver_interface=127.1.210.65
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:39019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.210.65
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.65.keytab with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:46:24.333117 10793 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:46:24.337229 10793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e: Bootstrap starting.
I20260806 02:46:24.358489 10793 log.cc:824] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e: Log is configured to *not* fsync() on all Append() calls
I20260806 02:46:24.367851 10793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:46:24.368471 10793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e: Bootstrap complete.
I20260806 02:46:24.378050 10793 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59bc1d3c57e44bc8ae33167f93d0a72e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 33657 } }
I20260806 02:46:24.379033 10793 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59bc1d3c57e44bc8ae33167f93d0a72e, State: Initialized, Role: FOLLOWER
I20260806 02:46:24.379786 10793 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59bc1d3c57e44bc8ae33167f93d0a72e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 33657 } }
I20260806 02:46:24.380054 10793 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:46:24.380185 10793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:46:24.380389 10793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:46:24.384843 10793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59bc1d3c57e44bc8ae33167f93d0a72e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 33657 } }
I20260806 02:46:24.385522 10793 leader_election.cc:302] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59bc1d3c57e44bc8ae33167f93d0a72e; no voters:
I20260806 02:46:24.386200 10793 leader_election.cc:288] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 02:46:24.386343 10797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e [term 2 FOLLOWER]: Leader election won for term 2
I20260806 02:46:24.387483 10797 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e [term 2 LEADER]: Becoming Leader. State: Replica: 59bc1d3c57e44bc8ae33167f93d0a72e, State: Running, Role: LEADER
I20260806 02:46:24.388162 10797 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59bc1d3c57e44bc8ae33167f93d0a72e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 33657 } }
I20260806 02:46:24.388507 10793 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:46:24.390813 10799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59bc1d3c57e44bc8ae33167f93d0a72e. Latest consensus state: current_term: 2 leader_uuid: "59bc1d3c57e44bc8ae33167f93d0a72e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59bc1d3c57e44bc8ae33167f93d0a72e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 33657 } } }
I20260806 02:46:24.391222 10799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e [sys.catalog]: This master's current role is: LEADER
I20260806 02:46:24.391732 10798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "59bc1d3c57e44bc8ae33167f93d0a72e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59bc1d3c57e44bc8ae33167f93d0a72e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 33657 } } }
I20260806 02:46:24.392043 10798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e [sys.catalog]: This master's current role is: LEADER
I20260806 02:46:24.397207 10803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:46:24.403409 10803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:46:24.404956 10803 catalog_manager.cc:1259] Loaded cluster ID: c97f78315a63419aa217100a6b728e67
I20260806 02:46:24.405141 10803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:46:24.409632 10803 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:46:24.411491 10803 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 59bc1d3c57e44bc8ae33167f93d0a72e: Loaded TSK: 0
I20260806 02:46:24.413391 10803 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 02:46:24.693778 10795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:24.694100 10795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:24.694265 10795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:24.702349 10795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:24.702483 10795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:24.702701 10795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.65
I20260806 02:46:24.713747 10795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:39019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.65.keytab
--principal=oryx/127.1.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.65:37937
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.65
--webserver_port=34559
--tserver_master_addrs=127.1.210.126:33657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.65
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:46:24.715333 10795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:24.717317 10795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:24.734767 10819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:24.734768 10820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:24.735001 10822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:24.736739 10795 server_base.cc:1034] running on GCE node
I20260806 02:46:24.737510 10795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:24.738866 10795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:24.740236 10795 hybrid_clock.cc:648] HybridClock initialized: now 1785984384740166 us; error 54 us; skew 500 ppm
I20260806 02:46:24.740690 10795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:46:24 dist-test-slave-8tdl krb5kdc[10199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984384, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:24.744745 10795 init.cc:377] Logged in from keytab as oryx/127.1.210.65@KRBTEST.COM (short username oryx)
I20260806 02:46:24.746574 10795 webserver.cc:460] Webserver started at http://127.1.210.65:34559/ using document root <none> and password file <none>
I20260806 02:46:24.747435 10795 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:24.747607 10795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:24.754240 10795 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:46:24.757702 10829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:24.759542 10795 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260806 02:46:24.759766 10795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-0/wal
uuid: "a0ec03f9f6254c838b83d6a4fdafe0b9"
format_stamp: "Formatted at 2026-08-06 02:46:21 on dist-test-slave-8tdl"
server_key: "b9966b9518db00634ff3ac3fabb4a05d"
server_key_iv: "2d059a89e3693b98b15e0a2b89484f83"
server_key_version: "encryptionkey@0"
I20260806 02:46:24.760807 10795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:24.782014 10795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:24.787110 10795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:24.787431 10795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:24.788614 10795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:46:24.790261 10795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:24.790396 10795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:24.790540 10795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:24.790642 10795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:24.826565 10795 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.65:37937
I20260806 02:46:24.826656 10943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.65:37937 every 8 connection(s)
I20260806 02:46:24.828109 10795 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
I20260806 02:46:24.834416 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 10795
I20260806 02:46:24.835160 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.66:44759
--local_ip_for_outbound_sockets=127.1.210.66
--tserver_master_addrs=127.1.210.126:33657
--webserver_port=42807
--webserver_interface=127.1.210.66
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:39019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.210.66
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.66.keytab with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:46:24 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:24 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
W20260806 02:46:25.192312 10948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:25.192623 10948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:25.192778 10948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:25.200699 10948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:25.200832 10948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:25.201007 10948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.66
I20260806 02:46:25.212832 10948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:39019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.66.keytab
--principal=oryx/127.1.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.66:44759
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.66
--webserver_port=42807
--tserver_master_addrs=127.1.210.126:33657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.66
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:46:25.214457 10948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:25.216094 10948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:25.236925 10954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:25.236677 10955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:25.238011 10957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:25.238864 10948 server_base.cc:1034] running on GCE node
I20260806 02:46:25.239750 10948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:25.241074 10948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:25.244536 10948 hybrid_clock.cc:648] HybridClock initialized: now 1785984385244411 us; error 97 us; skew 500 ppm
I20260806 02:46:25.245012 10948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:46:25 dist-test-slave-8tdl krb5kdc[10199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984385, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:25.248893 10948 init.cc:377] Logged in from keytab as oryx/127.1.210.66@KRBTEST.COM (short username oryx)
I20260806 02:46:25.251331 10948 webserver.cc:460] Webserver started at http://127.1.210.66:42807/ using document root <none> and password file <none>
I20260806 02:46:25.252146 10948 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:25.252310 10948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:25.259503 10948 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.003s
I20260806 02:46:25.263051 10964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:25.264813 10948 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:46:25.265013 10948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-1/wal
uuid: "cabcbf1a307741109d266cec6b5f40f6"
format_stamp: "Formatted at 2026-08-06 02:46:22 on dist-test-slave-8tdl"
server_key: "ac5e9d84f4d53628477d656e2d1babd6"
server_key_iv: "0f5800facc73943f6e5e575d0974b87c"
server_key_version: "encryptionkey@0"
I20260806 02:46:25.266115 10948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:25.284564 10948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:25.289865 10948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:25.290180 10948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:25.291450 10948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:46:25.293401 10948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:25.293512 10948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:25.293637 10948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:25.293716 10948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:25.345372 10948 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.66:44759
I20260806 02:46:25.345630 11079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.66:44759 every 8 connection(s)
I20260806 02:46:25.347582 10948 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
I20260806 02:46:25.353736 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 10948
I20260806 02:46:25.354552 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.67:34335
--local_ip_for_outbound_sockets=127.1.210.67
--tserver_master_addrs=127.1.210.126:33657
--webserver_port=44065
--webserver_interface=127.1.210.67
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:39019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.210.67
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.67.keytab with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:46:25 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:25 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
W20260806 02:46:25.727643 11084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:25.728144 11084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:25.728335 11084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:25.736440 11084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:25.736620 11084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:25.736817 11084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.67
I20260806 02:46:25.747289 11084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:39019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/krb5kdc/oryx_127.1.210.67.keytab
--principal=oryx/127.1.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.67:34335
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.67
--webserver_port=44065
--tserver_master_addrs=127.1.210.126:33657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.67
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:46:25.748929 11084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:25.750347 11084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:25.770411 11090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:25.775916 11091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:25.777411 11084 server_base.cc:1034] running on GCE node
W20260806 02:46:25.777700 11093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:25.778504 11084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:25.779789 11084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:25.781023 11084 hybrid_clock.cc:648] HybridClock initialized: now 1785984385780949 us; error 67 us; skew 500 ppm
I20260806 02:46:25.781404 11084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:46:25 dist-test-slave-8tdl krb5kdc[10199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984385, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:25.785063 11084 init.cc:377] Logged in from keytab as oryx/127.1.210.67@KRBTEST.COM (short username oryx)
I20260806 02:46:25.789076 11084 webserver.cc:460] Webserver started at http://127.1.210.67:44065/ using document root <none> and password file <none>
I20260806 02:46:25.790047 11084 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:25.790228 11084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:25.796290 11084 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260806 02:46:25.799856 11100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:25.801702 11084 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260806 02:46:25.801928 11084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-2/wal
uuid: "6e112115b2774648be5066e86a590308"
format_stamp: "Formatted at 2026-08-06 02:46:23 on dist-test-slave-8tdl"
server_key: "52e44744684f0cfc38c13d08d9862bd2"
server_key_iv: "ad9298d1a25a2e7a8e47772bd7d9c5c0"
server_key_version: "encryptionkey@0"
I20260806 02:46:25.803212 11084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:25.817248 11084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:25.822077 11084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:25.822389 11084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:25.823593 11084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:46:25.825152 11084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:25.825273 11084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:25.825418 11084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:25.825525 11084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:25.893291 11084 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.67:34335
I20260806 02:46:25.893399 11213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.67:34335 every 8 connection(s)
I20260806 02:46:25.896211 11084 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
I20260806 02:46:25.903393 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 11084
Aug 06 02:46:25 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:25 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:25.973721 11216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:25.905377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58914 (local address 127.1.210.126:33657)
0806 02:46:25.942234 (+ 36857us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:25.942245 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:25.942272 (+ 27us) server_negotiation.cc:407] Connection header received
0806 02:46:25.942355 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:25.942363 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:25.942504 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:25.942690 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:25.946211 (+ 3521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:25.948369 (+ 2158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:25.951516 (+ 3147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:25.952527 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:25.952708 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:25.958280 (+ 5572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:25.958301 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:25.958341 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:25.958433 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:25.964775 (+ 6342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:25.966297 (+ 1522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:25.966313 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:25.966328 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:25.966716 (+ 388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:25.967386 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:25.967426 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:25.967433 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:25.968495 (+ 1062us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:25.970640 (+ 2145us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:25.971072 (+ 432us) server_negotiation.cc:299] Negotiation successful
0806 02:46:25.972095 (+ 1023us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36585,"thread_start_us":16744,"threads_started":1}
W20260806 02:46:26.469583 10947 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.627s user 0.000s sys 0.014s
W20260806 02:46:26.472203 10946 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:24.831529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:48491 (local address 127.1.210.126:33657)
0806 02:46:24.832631 (+ 1102us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:24.832647 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:24.833065 (+ 418us) server_negotiation.cc:407] Connection header received
0806 02:46:24.835429 (+ 2364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:24.835449 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:24.835844 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:24.836214 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:24.837805 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:24.839153 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:24.840779 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:24.841514 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:24.841735 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:26.471682 (+1629947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.65:48491: BlockingRecv error: recv got EOF from 127.1.210.65:48491 (error 108)
Metrics: {"server-negotiator.queue_time_us":773,"thread_start_us":422,"threads_started":1}
I20260806 02:46:26.475548 10947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:24.831567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.65:48491)
0806 02:46:24.832874 (+ 1307us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:24.832913 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:24.834766 (+ 1853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:24.836383 (+ 1617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:24.836394 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:24.837042 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:24.837596 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:24.837614 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:24.839336 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:24.839341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:24.840603 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:24.840616 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:24.841779 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:24.841787 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:24.842100 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:24.842907 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:24.842931 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:26.473617 (+1630686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1056,"thread_start_us":848,"threads_started":1}
W20260806 02:46:26.477474 10944 heartbeater.cc:644] Failed to heartbeat to 127.1.210.126:33657 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.210.126:33657: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Aug 06 02:46:26 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:26 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:26.503607 10947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:26.478628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.65:48055)
0806 02:46:26.479348 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:26.479378 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:26.479630 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:26.480449 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:26.480457 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:26.481191 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:26.481528 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:26.481546 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:26.484334 (+ 2788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:26.484346 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:26.487339 (+ 2993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:26.487381 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:26.489294 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:26.489306 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:26.489798 (+ 492us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:26.495198 (+ 5400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:26.495247 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:26.503081 (+ 7834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250}
W20260806 02:46:26.503978 10946 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:26.478948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:48055 (local address 127.1.210.126:33657)
0806 02:46:26.479217 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:26.479228 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:26.479467 (+ 239us) server_negotiation.cc:407] Connection header received
0806 02:46:26.479847 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:26.479856 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:26.479979 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:26.480085 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:26.481817 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:26.483787 (+ 1970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:26.487694 (+ 3907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:26.488884 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:26.489166 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:26.503556 (+ 14390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.65:48055: BlockingRecv error: recv got EOF from 127.1.210.65:48055 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 06 02:46:26 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:26 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:26.525233 10947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:26.505437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.65:37827)
0806 02:46:26.505717 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:26.505751 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:26.506587 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:26.507560 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:26.507569 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:26.508359 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:26.508711 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:26.508730 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:26.511057 (+ 2327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:26.511068 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:26.513580 (+ 2512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:26.513609 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:26.515363 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:26.515375 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:26.515711 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:26.516822 (+ 1111us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:26.516861 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:26.524758 (+ 7897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":85}
W20260806 02:46:26.525609 10946 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:26.506102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:37827 (local address 127.1.210.126:33657)
0806 02:46:26.506364 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:26.506373 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:26.506405 (+ 32us) server_negotiation.cc:407] Connection header received
0806 02:46:26.506904 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:26.506991 (+ 87us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:26.507122 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:26.507259 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:26.508977 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:26.510794 (+ 1817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:26.513861 (+ 3067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:26.515079 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:26.515293 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:26.525171 (+ 9878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.65:37827: BlockingRecv error: recv got EOF from 127.1.210.65:37827 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
W20260806 02:46:26.528733 10944 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:46:26.725207 11083 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.356s user 0.002s sys 0.018s
W20260806 02:46:26.727082 11082 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:25.350859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:48701 (local address 127.1.210.126:33657)
0806 02:46:25.351427 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:25.351432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:25.356741 (+ 5309us) server_negotiation.cc:407] Connection header received
0806 02:46:25.356830 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:25.356838 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:25.356993 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:25.357120 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:25.360677 (+ 3557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:25.361931 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:25.365901 (+ 3970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:25.366753 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:25.366912 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:26.726518 (+1359606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.66:48701: BlockingRecv error: recv got EOF from 127.1.210.66:48701 (error 108)
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":284,"threads_started":1}
I20260806 02:46:26.727200 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:25.352005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.66:48701)
0806 02:46:25.353226 (+ 1221us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:25.353294 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:25.356270 (+ 2976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:25.357634 (+ 1364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:25.357655 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:25.358952 (+ 1297us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:25.360345 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:25.360386 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:25.362189 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:25.362198 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:25.365657 (+ 3459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:25.365684 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:25.367125 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:25.367136 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:25.367657 (+ 521us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:25.368791 (+ 1134us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:25.368842 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:26.726120 (+1357278us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":457,"threads_started":1}
W20260806 02:46:26.728440 11080 heartbeater.cc:644] Failed to heartbeat to 127.1.210.126:33657 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.210.126:33657: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Aug 06 02:46:26 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:26 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:26.740667 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:26.729367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.66:60659)
0806 02:46:26.729605 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:26.729630 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:26.729749 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:26.730175 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:26.730180 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:26.730568 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:26.730791 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:26.730802 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:26.731980 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:26.731985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:26.732946 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:26.732959 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:26.733756 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:26.733763 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:26.733944 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:26.734850 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:26.734879 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:26.740298 (+ 5419us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91}
W20260806 02:46:26.740803 11082 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:26.729671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:60659 (local address 127.1.210.126:33657)
0806 02:46:26.729898 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:26.729903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:26.729918 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:46:26.729966 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:26.729969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:26.730027 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:26.730138 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:26.730941 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:26.731742 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:26.733110 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:26.733558 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:26.733712 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:26.740545 (+ 6833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.66:60659: BlockingRecv error: recv got EOF from 127.1.210.66:60659 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 06 02:46:26 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:26 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:26.754325 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:26.741534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.66:50321)
0806 02:46:26.741697 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:26.741716 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:26.741850 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:26.742815 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:26.742819 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:26.743174 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:26.743363 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:26.743372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:26.745822 (+ 2450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:26.745827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:26.746957 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:26.746973 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:26.748394 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:26.748399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:26.748561 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:26.749435 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:26.749462 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:26.754060 (+ 4598us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 02:46:26.754796 11080 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:46:26.755039 11082 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:26.741993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:50321 (local address 127.1.210.126:33657)
0806 02:46:26.742248 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:26.742259 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:26.742287 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:46:26.742373 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:26.742382 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:26.742532 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:26.742749 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:26.743554 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:26.745568 (+ 2014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:26.747136 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:26.748177 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:26.748379 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:26.754561 (+ 6182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.66:50321: BlockingRecv error: recv got EOF from 127.1.210.66:50321 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260806 02:46:27.212204 11220 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.266s user 0.010s sys 0.017s
W20260806 02:46:27.213675 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:25.901007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:41533 (local address 127.1.210.126:33657)
0806 02:46:25.922529 (+ 21522us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:25.922540 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:25.922586 (+ 46us) server_negotiation.cc:407] Connection header received
0806 02:46:25.922713 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:25.922721 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:25.922853 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:25.927957 (+ 5104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:25.928363 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:25.937517 (+ 9154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:25.942065 (+ 4548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:25.943311 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:25.943540 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:27.213450 (+1269910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.67:41533: BlockingRecv error: recv got EOF from 127.1.210.67:41533 (error 108)
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":21292,"thread_start_us":40719,"threads_started":1}
I20260806 02:46:27.214109 11220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:25.902403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.67:41533)
0806 02:46:25.914800 (+ 12397us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:25.914915 (+ 115us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:25.918821 (+ 3906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:25.924680 (+ 5859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:25.924704 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:25.926051 (+ 1347us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:25.927612 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:25.927657 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:25.937965 (+ 10308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:25.937977 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:25.941446 (+ 3469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:25.941483 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:25.944058 (+ 2575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:25.944070 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:25.944604 (+ 534us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:25.945801 (+ 1197us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:25.945851 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:27.213093 (+1267242us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":11766,"thread_start_us":9919,"threads_started":1}
W20260806 02:46:27.215186 11214 heartbeater.cc:644] Failed to heartbeat to 127.1.210.126:33657 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.210.126:33657: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Aug 06 02:46:27 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:27 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:27.226128 11220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:27.215871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.67:33289)
0806 02:46:27.216157 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:27.216175 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:27.216278 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:27.216604 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:27.216607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:27.216966 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:27.217199 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:27.217208 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:27.218606 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:27.218612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:27.220060 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:27.220077 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:27.220882 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:27.220887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:27.221031 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:27.221731 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:27.221751 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:27.225862 (+ 4111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100}
W20260806 02:46:27.226516 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:27.216086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:33289 (local address 127.1.210.126:33657)
0806 02:46:27.216262 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:27.216267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:27.216291 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:46:27.216386 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:27.216390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:27.216452 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:27.216547 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:27.217383 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:27.218364 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:27.220236 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:27.220697 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:27.220807 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:27.226252 (+ 5445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.67:33289: BlockingRecv error: recv got EOF from 127.1.210.67:33289 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 06 02:46:27 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:27 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:27.236421 11220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:27.227019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.67:43391)
0806 02:46:27.227200 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:27.227217 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:27.227359 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:27.227781 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:27.227785 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:27.228224 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:27.228392 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:27.228402 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:27.229464 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:27.229476 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:27.230622 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:27.230639 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:27.231656 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:27.231662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:27.231857 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:27.232561 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:27.232581 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:27.236193 (+ 3612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260806 02:46:27.236732 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:27.227133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:43391 (local address 127.1.210.126:33657)
0806 02:46:27.227477 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:27.227483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:27.227510 (+ 27us) server_negotiation.cc:407] Connection header received
0806 02:46:27.227562 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:27.227565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:27.227628 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:27.227740 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:27.228547 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:27.229305 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:27.230846 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:27.231491 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:27.231637 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:27.236446 (+ 4809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.67:43391: BlockingRecv error: recv got EOF from 127.1.210.67:43391 (error 108)
Metrics: {"server-negotiator.queue_time_us":175}
W20260806 02:46:27.236816 11214 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 06 02:46:27 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:27 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:27.539778 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:27.529438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.65:45791)
0806 02:46:27.530229 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:27.530251 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:27.530408 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:27.530831 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:27.530836 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:27.531312 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:27.531463 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:27.531472 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:27.532509 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:27.532516 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:27.533508 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:27.533521 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:27.534290 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:27.534296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:27.534488 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:27.535207 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:27.535226 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:27.539351 (+ 4125us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":572,"threads_started":1}
W20260806 02:46:27.539837 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:27.529540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:45791 (local address 127.1.210.126:33657)
0806 02:46:27.529817 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:27.529827 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:27.530334 (+ 507us) server_negotiation.cc:407] Connection header received
0806 02:46:27.530547 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:27.530552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:27.530650 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:27.530788 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:27.531636 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:27.532359 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:27.533671 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:27.534099 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:27.534236 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:27.539571 (+ 5335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.65:45791: BlockingRecv error: recv got EOF from 127.1.210.65:45791 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Aug 06 02:46:27 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:27 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:27.766088 11223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:27.755515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.66:46909)
0806 02:46:27.756212 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:27.756229 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:27.756406 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:27.756988 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:27.756994 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:27.757621 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:27.757807 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:27.757821 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:27.759282 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:27.759286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:27.760282 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:27.760296 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:27.761051 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:27.761056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:27.761180 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:27.761858 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:27.761877 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:27.765739 (+ 3862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":404,"threads_started":1}
W20260806 02:46:27.766309 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:27.755704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:46909 (local address 127.1.210.126:33657)
0806 02:46:27.755972 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:27.755982 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:27.756328 (+ 346us) server_negotiation.cc:407] Connection header received
0806 02:46:27.756553 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:27.756565 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:27.756669 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:27.756779 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:27.758051 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:27.759133 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:27.760463 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:27.760881 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:27.760996 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:27.766027 (+ 5031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.66:46909: BlockingRecv error: recv got EOF from 127.1.210.66:46909 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Aug 06 02:46:28 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:28 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:28.248066 11224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:28.237525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.67:53061)
0806 02:46:28.238207 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:28.238228 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:28.238348 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:28.238717 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:28.238721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:28.239201 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:28.239355 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:28.239364 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:28.240331 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:28.240335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:28.241488 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:28.241506 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:28.242726 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:28.242737 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:28.242892 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:28.243828 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:28.243847 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:28.247766 (+ 3919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":371,"threads_started":1}
W20260806 02:46:28.248371 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:28.237622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:53061 (local address 127.1.210.126:33657)
0806 02:46:28.237795 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:28.237800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:28.238324 (+ 524us) server_negotiation.cc:407] Connection header received
0806 02:46:28.238458 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:28.238462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:28.238545 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:28.238644 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:28.239488 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:28.240210 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:28.241656 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:28.242417 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:28.242591 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:28.248012 (+ 5421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.67:53061: BlockingRecv error: recv got EOF from 127.1.210.67:53061 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 06 02:46:28 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:28 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:28.551792 11225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:28.540961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.65:53913)
0806 02:46:28.541541 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:28.541560 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:28.541700 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:28.542130 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:28.542133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:28.542528 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:28.542742 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:28.542751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:28.543931 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:28.543935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:28.544948 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:28.544964 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:28.546133 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:28.546140 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:28.546331 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:28.547396 (+ 1065us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:28.547426 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:28.551489 (+ 4063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":328,"threads_started":1}
W20260806 02:46:28.552122 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:28.541193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:53913 (local address 127.1.210.126:33657)
0806 02:46:28.541392 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:28.541397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:28.541652 (+ 255us) server_negotiation.cc:407] Connection header received
0806 02:46:28.541814 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:28.541818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:28.541887 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:28.542043 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:28.542970 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:28.543749 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:28.545146 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:28.545830 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:28.546094 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:28.551755 (+ 5661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.65:53913: BlockingRecv error: recv got EOF from 127.1.210.65:53913 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 06 02:46:28 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:28 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:28.776702 11226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:28.767213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.66:58379)
0806 02:46:28.767717 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:28.767734 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:28.767864 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:28.768193 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:28.768196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:28.768623 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:28.768808 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:28.768816 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:28.769954 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:28.769959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:28.771060 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:28.771074 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:28.771790 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:28.771795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:28.771947 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:28.772543 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:28.772562 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:28.776322 (+ 3760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":311,"threads_started":1}
W20260806 02:46:28.776786 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:28.767302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:58379 (local address 127.1.210.126:33657)
0806 02:46:28.767562 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:28.767569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:28.767830 (+ 261us) server_negotiation.cc:407] Connection header received
0806 02:46:28.767972 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:28.767976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:28.768055 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:28.768144 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:28.768985 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:28.769721 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:28.771196 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:28.771616 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:28.771724 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:28.776548 (+ 4824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.66:58379: BlockingRecv error: recv got EOF from 127.1.210.66:58379 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 06 02:46:29 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:29 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:29.261546 11227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:29.249255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.67:41449)
0806 02:46:29.249824 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:29.249849 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:29.250049 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:29.250532 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:29.250537 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:29.251264 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:29.251476 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:29.251486 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:29.252721 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:29.252725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:29.253816 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:29.253835 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:29.254851 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:29.254857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:29.255027 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:29.256132 (+ 1105us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:29.256166 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:29.261093 (+ 4927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":318,"threads_started":1}
W20260806 02:46:29.261637 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:29.249513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:41449 (local address 127.1.210.126:33657)
0806 02:46:29.249763 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:29.249771 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:29.250118 (+ 347us) server_negotiation.cc:407] Connection header received
0806 02:46:29.250183 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:29.250187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:29.250265 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:29.250356 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:29.251710 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:29.252497 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:29.254066 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:29.254536 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:29.254650 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:29.261369 (+ 6719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.67:41449: BlockingRecv error: recv got EOF from 127.1.210.67:41449 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Aug 06 02:46:29 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:29 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:29.564680 11228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:29.553060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.65:35891)
0806 02:46:29.553722 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:29.553739 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:29.553871 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:29.554329 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:29.554333 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:29.554831 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:29.555067 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:29.555078 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:29.556389 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:29.556394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:29.557676 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:29.557693 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:29.558654 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:29.558659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:29.558899 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:29.559712 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:29.559731 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:29.564317 (+ 4586us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":358,"threads_started":1}
W20260806 02:46:29.565032 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:29.553339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:35891 (local address 127.1.210.126:33657)
0806 02:46:29.553532 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:29.553538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:29.553822 (+ 284us) server_negotiation.cc:407] Connection header received
0806 02:46:29.554072 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:29.554077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:29.554148 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:29.554244 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:29.555272 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:29.556214 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:29.557864 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:29.558448 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:29.558622 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:29.564689 (+ 6067us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.65:35891: BlockingRecv error: recv got EOF from 127.1.210.65:35891 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 06 02:46:29 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:29 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:29.789717 11229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:29.777958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.66:43223)
0806 02:46:29.778574 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:29.778591 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:29.778812 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:29.779358 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:29.779362 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:29.780142 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:29.780402 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:29.780413 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:29.781985 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:29.782013 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:29.783118 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:29.783133 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:29.784006 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:29.784012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:29.784176 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:29.784915 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:29.784935 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:29.789432 (+ 4497us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":341,"threads_started":1}
W20260806 02:46:29.789978 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:29.778122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:43223 (local address 127.1.210.126:33657)
0806 02:46:29.778413 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:29.778418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:29.778766 (+ 348us) server_negotiation.cc:407] Connection header received
0806 02:46:29.779092 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:29.779097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:29.779169 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:29.779305 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:29.780611 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:29.781773 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:29.783284 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:29.783784 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:29.783891 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:29.789689 (+ 5798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.66:43223: BlockingRecv error: recv got EOF from 127.1.210.66:43223 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 06 02:46:30 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:30 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:30.273859 11230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:30.263166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.67:53163)
0806 02:46:30.263738 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:30.263762 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:30.263969 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:30.264403 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:30.264408 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:30.264868 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:30.265204 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:30.265217 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:30.266390 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:30.266397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:30.267531 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:30.267545 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:30.268352 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:30.268359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:30.268576 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:30.269306 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:30.269326 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:30.273597 (+ 4271us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":292,"threads_started":1}
W20260806 02:46:30.274142 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:30.263407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:53163 (local address 127.1.210.126:33657)
0806 02:46:30.263646 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:30.263652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:30.263898 (+ 246us) server_negotiation.cc:407] Connection header received
0806 02:46:30.264146 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:30.264152 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:30.264252 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:30.264370 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:30.265376 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:30.266212 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:30.267698 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:30.268180 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:30.268358 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:30.273861 (+ 5503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.67:53163: BlockingRecv error: recv got EOF from 127.1.210.67:53163 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 06 02:46:30 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:30 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:30.576398 11231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:30.565904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.65:37355)
0806 02:46:30.566473 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:30.566491 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:30.566616 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:30.567011 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:30.567015 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:30.567418 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:30.567629 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:30.567638 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:30.568752 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:30.568757 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:30.569814 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:30.569828 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:30.570795 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:30.570800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:30.570951 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:30.571652 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:30.571680 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:30.576103 (+ 4423us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":292,"threads_started":1}
W20260806 02:46:30.576658 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:30.566040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:37355 (local address 127.1.210.126:33657)
0806 02:46:30.566336 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:30.566343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:30.566584 (+ 241us) server_negotiation.cc:407] Connection header received
0806 02:46:30.566752 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:30.566756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:30.566861 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:30.566966 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:30.567796 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:30.568585 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:30.570114 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:30.570563 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:30.570728 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:30.576337 (+ 5609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.65:37355: BlockingRecv error: recv got EOF from 127.1.210.65:37355 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Aug 06 02:46:30 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:30 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:30.801923 11232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:30.791060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.66:58323)
0806 02:46:30.791651 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:30.791668 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:30.791845 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:30.792284 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:30.792288 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:30.792759 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:30.792990 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:30.793001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:30.794331 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:30.794337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:30.795492 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:30.795506 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:30.796322 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:30.796328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:30.796516 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:30.797177 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:30.797196 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:30.801544 (+ 4348us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":317,"threads_started":1}
W20260806 02:46:30.802069 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:30.791160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:58323 (local address 127.1.210.126:33657)
0806 02:46:30.791451 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:30.791458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:30.791772 (+ 314us) server_negotiation.cc:407] Connection header received
0806 02:46:30.792025 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:30.792031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:30.792120 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:30.792233 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:30.793141 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:30.793937 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:30.795649 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:30.796152 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:30.796293 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:30.801827 (+ 5534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.66:58323: BlockingRecv error: recv got EOF from 127.1.210.66:58323 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Aug 06 02:46:31 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:31 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:31.285010 11233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:31.275092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.67:60581)
0806 02:46:31.275698 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:31.275727 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:31.275881 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:31.276262 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:31.276267 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:31.276678 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:31.276837 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:31.276846 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:31.277935 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:31.277941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:31.278984 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:31.278998 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:31.279701 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:31.279707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:31.279884 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:31.280486 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:31.280504 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:31.284684 (+ 4180us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":320,"threads_started":1}
W20260806 02:46:31.285161 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:31.275357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:60581 (local address 127.1.210.126:33657)
0806 02:46:31.275654 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:31.275661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:31.275812 (+ 151us) server_negotiation.cc:407] Connection header received
0806 02:46:31.276012 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:31.276026 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:31.276111 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:31.276219 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:31.277005 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:31.277785 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:31.279127 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:31.279550 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:31.279688 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:31.284902 (+ 5214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.67:60581: BlockingRecv error: recv got EOF from 127.1.210.67:60581 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 06 02:46:31 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:31 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:31.593825 11234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:31.577736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.65:57539)
0806 02:46:31.578323 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:31.578340 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:31.578550 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:31.578984 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:31.578990 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:31.579676 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:31.579973 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:31.579987 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:31.581306 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:31.581311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:31.582529 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:31.582545 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:31.583427 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:31.583433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:31.583578 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:31.589049 (+ 5471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:31.589072 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:31.593578 (+ 4506us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":320,"threads_started":1}
W20260806 02:46:31.594210 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:31.577820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:57539 (local address 127.1.210.126:33657)
0806 02:46:31.578091 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:31.578098 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:31.578482 (+ 384us) server_negotiation.cc:407] Connection header received
0806 02:46:31.578738 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:31.578747 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:31.578819 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:31.578907 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:31.580189 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:31.581011 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:31.582750 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:31.583214 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:31.583321 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:31.593853 (+ 10532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.65:57539: BlockingRecv error: recv got EOF from 127.1.210.65:57539 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 06 02:46:31 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:31 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:31.813683 11235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:31.803114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.66:43561)
0806 02:46:31.803766 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:31.803788 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:31.803930 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:31.804324 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:31.804328 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:31.804787 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:31.804953 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:31.804962 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:31.806143 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:31.806148 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:31.807468 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:31.807485 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:31.808557 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:31.808565 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:31.808785 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:31.809617 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:31.809641 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:31.813303 (+ 3662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":363,"threads_started":1}
W20260806 02:46:31.813819 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:31.803363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:43561 (local address 127.1.210.126:33657)
0806 02:46:31.803563 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:31.803568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:31.803885 (+ 317us) server_negotiation.cc:407] Connection header received
0806 02:46:31.804051 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:31.804055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:31.804130 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:31.804236 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:31.805149 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:31.805970 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:31.807652 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:31.808385 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:31.808507 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:31.813547 (+ 5040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.66:43561: BlockingRecv error: recv got EOF from 127.1.210.66:43561 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 06 02:46:32 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:32 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:32.297082 11236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:32.286243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.67:47825)
0806 02:46:32.286992 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:32.287015 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:32.287194 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:32.287648 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:32.287653 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:32.288150 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:32.288307 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:32.288317 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:32.289608 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:32.289614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:32.290792 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:32.290806 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:32.291663 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:32.291670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:32.291887 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:32.292643 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:32.292664 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:32.296714 (+ 4050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":461,"threads_started":1}
W20260806 02:46:32.297279 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:32.286420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:47825 (local address 127.1.210.126:33657)
0806 02:46:32.286742 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:32.286750 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:32.287151 (+ 401us) server_negotiation.cc:407] Connection header received
0806 02:46:32.287365 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:32.287369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:32.287459 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:32.287583 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:32.288467 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:32.289436 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:32.290963 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:32.291478 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:32.291644 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:32.296994 (+ 5350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.67:47825: BlockingRecv error: recv got EOF from 127.1.210.67:47825 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Aug 06 02:46:32 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:32 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:32.605927 11237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:32.595462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.65:57411)
0806 02:46:32.596117 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:32.596134 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:32.596306 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:32.596700 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:32.596704 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:32.597170 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:32.597329 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:32.597338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:32.598519 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:32.598527 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:32.599588 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:32.599602 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:32.600369 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:32.600376 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:32.600562 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:32.601413 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:32.601433 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:32.605560 (+ 4127us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":341,"threads_started":1}
W20260806 02:46:32.606240 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:32.595599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:57411 (local address 127.1.210.126:33657)
0806 02:46:32.595896 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:32.595902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:32.596228 (+ 326us) server_negotiation.cc:407] Connection header received
0806 02:46:32.596486 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:32.596490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:32.596555 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:32.596665 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:32.597483 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:32.598364 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:32.599750 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:32.600198 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:32.600336 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:32.605808 (+ 5472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.65:57411: BlockingRecv error: recv got EOF from 127.1.210.65:57411 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Aug 06 02:46:32 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:32 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:32.825171 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:32.814827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.66:51877)
0806 02:46:32.815419 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:32.815437 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:32.815572 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:32.815936 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:32.815939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:32.816538 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:32.816720 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:32.816731 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:32.817811 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:32.817816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:32.818846 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:32.818858 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:32.819623 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:32.819628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:32.819762 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:32.820382 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:32.820401 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:32.824939 (+ 4538us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":354,"threads_started":1}
W20260806 02:46:32.825418 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:32.814924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:51877 (local address 127.1.210.126:33657)
0806 02:46:32.815194 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:32.815205 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:32.815554 (+ 349us) server_negotiation.cc:407] Connection header received
0806 02:46:32.815701 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:32.815705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:32.815790 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:32.815884 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:32.816893 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:32.817652 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:32.818984 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:32.819440 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:32.819542 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:32.825146 (+ 5604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.66:51877: BlockingRecv error: recv got EOF from 127.1.210.66:51877 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 06 02:46:33 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:33 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:33.309057 11239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:33.298282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.67:59917)
0806 02:46:33.298866 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:33.298888 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:33.299054 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:33.299457 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:33.299462 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:33.299873 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:33.300123 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:33.300133 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:33.301294 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:33.301298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:33.302295 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:33.302311 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:33.303120 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:33.303125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:33.303366 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:33.304149 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:33.304170 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:33.308676 (+ 4506us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":328,"threads_started":1}
W20260806 02:46:33.309230 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:33.298315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:59917 (local address 127.1.210.126:33657)
0806 02:46:33.298628 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:33.298636 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:33.299088 (+ 452us) server_negotiation.cc:407] Connection header received
0806 02:46:33.299190 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:33.299193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:33.299277 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:33.299384 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:33.300297 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:33.301131 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:33.302474 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:33.302910 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:33.303038 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:33.308920 (+ 5882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.67:59917: BlockingRecv error: recv got EOF from 127.1.210.67:59917 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 06 02:46:33 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:33 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:33.621016 11240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:33.607483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.65:47795)
0806 02:46:33.608540 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:33.608559 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:33.608720 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:33.609425 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:33.609431 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:33.610031 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:33.610249 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:33.610261 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:33.611771 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:33.611777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:33.613225 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:33.613243 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:33.614184 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:33.614190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:33.614316 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:33.615101 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:33.615122 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:33.620586 (+ 5464us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":928,"thread_start_us":653,"threads_started":1}
W20260806 02:46:33.621167 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:33.607613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:47795 (local address 127.1.210.126:33657)
0806 02:46:33.607880 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:33.607891 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:33.608697 (+ 806us) server_negotiation.cc:407] Connection header received
0806 02:46:33.608859 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:33.608864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:33.608943 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:33.609071 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:33.610424 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:33.611480 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:33.613451 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:33.613958 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:33.614083 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:33.620912 (+ 6829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.65:47795: BlockingRecv error: recv got EOF from 127.1.210.65:47795 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 06 02:46:33 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:33 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:33.836813 11241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:33.826317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.66:39459)
0806 02:46:33.826981 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:33.827000 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:33.827129 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:33.827639 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:33.827643 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:33.828118 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:33.828327 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:33.828336 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:33.829881 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:33.829886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:33.830969 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:33.830983 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:33.831771 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:33.831775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:33.831895 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:33.832566 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:33.832584 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:33.836496 (+ 3912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":361,"threads_started":1}
W20260806 02:46:33.837126 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:33.826385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:39459 (local address 127.1.210.126:33657)
0806 02:46:33.826759 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:33.826769 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:33.827149 (+ 380us) server_negotiation.cc:407] Connection header received
0806 02:46:33.827327 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:33.827332 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:33.827411 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:33.827561 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:33.828508 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:33.829706 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:33.831140 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:33.831609 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:33.831713 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:33.836807 (+ 5094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.66:39459: BlockingRecv error: recv got EOF from 127.1.210.66:39459 (error 108)
Metrics: {"server-negotiator.queue_time_us":137}
Aug 06 02:46:34 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:34 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:34.320284 11242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:34.310058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.67:38975)
0806 02:46:34.310876 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:34.310897 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:34.311018 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:34.311492 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:34.311496 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:34.311938 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:34.312131 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:34.312142 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:34.313277 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:34.313282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:34.314234 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:34.314248 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:34.315091 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:34.315096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:34.315226 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:34.315896 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:34.315914 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:34.319862 (+ 3948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":370,"threads_started":1}
W20260806 02:46:34.320325 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:34.310142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:38975 (local address 127.1.210.126:33657)
0806 02:46:34.310376 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:34.310386 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:34.310966 (+ 580us) server_negotiation.cc:407] Connection header received
0806 02:46:34.311149 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:34.311158 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:34.311277 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:34.311401 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:34.312283 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:34.313124 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:34.314428 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:34.314937 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:34.315042 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:34.320105 (+ 5063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.67:38975: BlockingRecv error: recv got EOF from 127.1.210.67:38975 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 06 02:46:34 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:34 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:34.632869 11243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:34.622204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.65:38119)
0806 02:46:34.622816 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:34.622834 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:34.622969 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:34.623359 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:34.623363 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:34.623800 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:34.623989 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:34.623998 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:34.625146 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:34.625151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:34.626240 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:34.626256 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:34.627187 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:34.627192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:34.627330 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:34.628067 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:34.628093 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:34.632599 (+ 4506us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":258,"threads_started":1}
W20260806 02:46:34.633116 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:34.622491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:38119 (local address 127.1.210.126:33657)
0806 02:46:34.622779 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:34.622784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:34.622921 (+ 137us) server_negotiation.cc:407] Connection header received
0806 02:46:34.623092 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:34.623095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:34.623180 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:34.623268 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:34.624169 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:34.624944 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:34.626458 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:34.626975 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:34.627096 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:34.632838 (+ 5742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.65:38119: BlockingRecv error: recv got EOF from 127.1.210.65:38119 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Aug 06 02:46:34 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:34 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:34.849359 11244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:34.837928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.66:40517)
0806 02:46:34.838583 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:34.838602 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:34.838800 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:34.839387 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:34.839390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:34.839858 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:34.840050 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:34.840060 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:34.841778 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:34.841783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:34.843141 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:34.843159 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:34.844078 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:34.844084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:34.844242 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:34.844929 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:34.844949 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:34.848934 (+ 3985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":356,"threads_started":1}
W20260806 02:46:34.849545 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:34.838151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:40517 (local address 127.1.210.126:33657)
0806 02:46:34.838347 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:34.838352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:34.838721 (+ 369us) server_negotiation.cc:407] Connection header received
0806 02:46:34.839010 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:34.839019 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:34.839125 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:34.839254 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:34.840202 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:34.841597 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:34.843314 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:34.843801 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:34.843913 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:34.849277 (+ 5364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.66:40517: BlockingRecv error: recv got EOF from 127.1.210.66:40517 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 06 02:46:35 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:35 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:35.333500 11245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:35.321685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.67:44671)
0806 02:46:35.322345 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:35.322373 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:35.322545 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:35.322932 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:35.322937 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:35.323491 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:35.323706 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:35.323717 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:35.325022 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:35.325028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:35.326879 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:35.326896 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:35.328018 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:35.328024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:35.328176 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:35.328876 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:35.328895 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:35.333220 (+ 4325us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":303,"threads_started":1}
W20260806 02:46:35.333792 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:35.321793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:44671 (local address 127.1.210.126:33657)
0806 02:46:35.322054 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:35.322062 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:35.322481 (+ 419us) server_negotiation.cc:407] Connection header received
0806 02:46:35.322697 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:35.322702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:35.322770 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:35.322870 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:35.323935 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:35.324774 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:35.327092 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:35.327812 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:35.327938 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:35.333521 (+ 5583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.67:44671: BlockingRecv error: recv got EOF from 127.1.210.67:44671 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 06 02:46:35 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:35 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:35.645893 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:35.634013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.65:48855)
0806 02:46:35.634573 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:35.634590 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:35.634773 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:35.635158 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:35.635162 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:35.635630 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:35.635791 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:35.635799 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:35.637246 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:35.637252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:35.638381 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:35.638396 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:35.639261 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:35.639268 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:35.639485 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:35.640324 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:35.640345 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:35.645367 (+ 5022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":285,"threads_started":1}
W20260806 02:46:35.646070 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:35.634167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:48855 (local address 127.1.210.126:33657)
0806 02:46:35.634375 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:35.634380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:35.634690 (+ 310us) server_negotiation.cc:407] Connection header received
0806 02:46:35.634913 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:35.634917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:35.634984 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:35.635113 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:35.635983 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:35.637040 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:35.638564 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:35.639063 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:35.639227 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:35.645793 (+ 6566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.65:48855: BlockingRecv error: recv got EOF from 127.1.210.65:48855 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 06 02:46:35 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:35 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:35.860360 11247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:35.850805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.66:48439)
0806 02:46:35.851413 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:35.851430 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:35.851632 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:35.851998 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:35.852003 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:35.852454 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:35.852606 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:35.852615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:35.853703 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:35.853709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:35.854719 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:35.854732 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:35.855512 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:35.855518 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:35.855712 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:35.856379 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:35.856398 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:35.860123 (+ 3725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":320,"threads_started":1}
W20260806 02:46:35.860857 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:35.850855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:48439 (local address 127.1.210.126:33657)
0806 02:46:35.851157 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:35.851166 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:35.851538 (+ 372us) server_negotiation.cc:407] Connection header received
0806 02:46:35.851791 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:35.851795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:35.851859 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:35.851966 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:35.852761 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:35.853557 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:35.854888 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:35.855351 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:35.855495 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:35.860449 (+ 4954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.66:48439: BlockingRecv error: recv got EOF from 127.1.210.66:48439 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 06 02:46:36 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:36 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:36.344959 11248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:36.334811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.67:44561)
0806 02:46:36.335408 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:36.335430 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:36.335551 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:36.335845 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:36.335848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:36.336362 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:36.336527 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:36.336537 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:36.337632 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:36.337636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:36.338748 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:36.338771 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:36.339578 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:36.339583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:36.339735 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:36.340498 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:36.340517 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:36.344606 (+ 4089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":313,"threads_started":1}
W20260806 02:46:36.345140 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:36.334860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:44561 (local address 127.1.210.126:33657)
0806 02:46:36.335064 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:36.335069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:36.335510 (+ 441us) server_negotiation.cc:407] Connection header received
0806 02:46:36.335639 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:36.335643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:36.335706 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:36.335795 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:36.336714 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:36.337472 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:36.338924 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:36.339400 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:36.339508 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:36.344868 (+ 5360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.67:44561: BlockingRecv error: recv got EOF from 127.1.210.67:44561 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 06 02:46:36 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:36 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:36.671175 11249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:36.647152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.65:43653)
0806 02:46:36.647713 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:36.647733 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:36.647858 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:36.648337 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:36.648340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:36.648721 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:36.648928 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:36.648938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:36.650136 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:36.650141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:36.651239 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:36.651253 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:36.652100 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:36.652105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:36.652231 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:36.666020 (+ 13789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:36.666050 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:36.670775 (+ 4725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":271,"threads_started":1}
W20260806 02:46:36.671468 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:36.647459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:43653 (local address 127.1.210.126:33657)
0806 02:46:36.647718 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:36.647724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:36.647800 (+ 76us) server_negotiation.cc:407] Connection header received
0806 02:46:36.648080 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:36.648085 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:36.648155 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:36.648251 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:36.649155 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:36.649954 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:36.651403 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:36.651848 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:36.651960 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:36.671154 (+ 19194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.65:43653: BlockingRecv error: recv got EOF from 127.1.210.65:43653 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Aug 06 02:46:36 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:36 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:36.872219 11250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:36.861536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.66:53561)
0806 02:46:36.862145 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:36.862169 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:36.862341 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:36.862745 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:36.862750 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:36.863148 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:36.863302 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:36.863310 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:36.864485 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:36.864490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:36.865531 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:36.865544 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:36.866267 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:36.866274 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:36.866462 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:36.867233 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:36.867254 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:36.871832 (+ 4578us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":398,"threads_started":1}
W20260806 02:46:36.872476 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:36.861691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:53561 (local address 127.1.210.126:33657)
0806 02:46:36.861922 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:36.861929 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:36.862323 (+ 394us) server_negotiation.cc:407] Connection header received
0806 02:46:36.862481 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:36.862485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:36.862552 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:36.862654 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:36.863466 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:36.864315 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:36.865677 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:36.866102 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:36.866249 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:36.872158 (+ 5909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.66:53561: BlockingRecv error: recv got EOF from 127.1.210.66:53561 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 06 02:46:37 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:37 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:37.357952 11251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:37.346277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.67:56429)
0806 02:46:37.347004 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:37.347027 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:37.347156 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:37.347552 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:37.347557 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:37.347962 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:37.348157 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:37.348168 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:37.349835 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:37.349840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:37.351506 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:37.351530 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:37.352599 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:37.352607 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:37.352809 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:37.353504 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:37.353523 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:37.357555 (+ 4032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":462,"threads_started":1}
W20260806 02:46:37.358137 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:37.346493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:56429 (local address 127.1.210.126:33657)
0806 02:46:37.346778 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:37.346786 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:37.347142 (+ 356us) server_negotiation.cc:407] Connection header received
0806 02:46:37.347278 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:37.347285 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:37.347372 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:37.347495 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:37.348343 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:37.349619 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:37.351703 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:37.352411 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:37.352571 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:37.357821 (+ 5250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.67:56429: BlockingRecv error: recv got EOF from 127.1.210.67:56429 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
Aug 06 02:46:37 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:37 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:37.683131 11252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:37.672333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.65:55329)
0806 02:46:37.672962 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:37.672980 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:37.673124 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:37.673570 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:37.673575 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:37.674063 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:37.674256 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:37.674267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:37.675613 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:37.675620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:37.676961 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:37.676998 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:37.677802 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:37.677807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:37.677933 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:37.678646 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:37.678665 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:37.682889 (+ 4224us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":351,"threads_started":1}
W20260806 02:46:37.683493 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:37.672447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:55329 (local address 127.1.210.126:33657)
0806 02:46:37.672673 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:37.672678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:37.673112 (+ 434us) server_negotiation.cc:407] Connection header received
0806 02:46:37.673280 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:37.673285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:37.673361 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:37.673495 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:37.674463 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:37.675439 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:37.677156 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:37.677636 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:37.677748 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:37.683224 (+ 5476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.65:55329: BlockingRecv error: recv got EOF from 127.1.210.65:55329 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 06 02:46:37 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:37 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:37.883890 11253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:37.873357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.66:48841)
0806 02:46:37.873949 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:37.873965 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:37.874174 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:37.874581 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:37.874588 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:37.875115 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:37.875298 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:37.875308 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:37.876443 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:37.876449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:37.877524 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:37.877539 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:37.878298 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:37.878303 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:37.878494 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:37.879219 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:37.879239 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:37.883531 (+ 4292us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":317,"threads_started":1}
W20260806 02:46:37.884012 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:37.873546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:48841 (local address 127.1.210.126:33657)
0806 02:46:37.873815 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:37.873822 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:37.874061 (+ 239us) server_negotiation.cc:407] Connection header received
0806 02:46:37.874348 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:37.874352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:37.874422 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:37.874534 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:37.875457 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:37.876293 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:37.877693 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:37.878127 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:37.878269 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:37.883743 (+ 5474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.66:48841: BlockingRecv error: recv got EOF from 127.1.210.66:48841 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Aug 06 02:46:38 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:38 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:38.370481 11254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:38.359213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.67:48811)
0806 02:46:38.359812 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:38.359838 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:38.360029 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:38.360413 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:38.360417 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:38.360886 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:38.361231 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:38.361246 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:38.362671 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:38.362743 (+ 72us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:38.363814 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:38.363829 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:38.364659 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:38.364668 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:38.364860 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:38.365521 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:38.365541 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:38.370239 (+ 4698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":344,"threads_started":1}
W20260806 02:46:38.370781 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:38.359344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:48811 (local address 127.1.210.126:33657)
0806 02:46:38.359750 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:38.359759 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:38.360013 (+ 254us) server_negotiation.cc:407] Connection header received
0806 02:46:38.360152 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:38.360156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:38.360226 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:38.360327 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:38.361452 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:38.362524 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:38.363984 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:38.364437 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:38.364618 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:38.370465 (+ 5847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.67:48811: BlockingRecv error: recv got EOF from 127.1.210.67:48811 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 06 02:46:38 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:38 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:38.695832 11255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:38.684327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.65:56809)
0806 02:46:38.684933 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:38.684957 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:38.685147 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:38.685526 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:38.685531 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:38.686118 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:38.686328 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:38.686338 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:38.687505 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:38.687510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:38.688623 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:38.688639 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:38.689769 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:38.689774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:38.689993 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:38.690765 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:38.690788 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:38.695534 (+ 4746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":305,"threads_started":1}
W20260806 02:46:38.696178 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:38.684428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:56809 (local address 127.1.210.126:33657)
0806 02:46:38.684731 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:38.684740 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:38.685158 (+ 418us) server_negotiation.cc:407] Connection header received
0806 02:46:38.685258 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:38.685262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:38.685350 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:38.685455 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:38.686492 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:38.687338 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:38.688814 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:38.689589 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:38.689719 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:38.695799 (+ 6080us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.65:56809: BlockingRecv error: recv got EOF from 127.1.210.65:56809 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 06 02:46:38 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:38 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:38.896358 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:38.885372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.66:45849)
0806 02:46:38.885920 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:38.886001 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:38.886197 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:38.886602 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:38.886607 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:38.887078 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:38.887257 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:38.887266 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:38.888325 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:38.888331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:38.889323 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:38.889335 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:38.890088 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:38.890094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:38.890293 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:38.891148 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:38.891172 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:38.895964 (+ 4792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":322,"threads_started":1}
W20260806 02:46:38.896392 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:38.885370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:45849 (local address 127.1.210.126:33657)
0806 02:46:38.885701 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:38.885709 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:38.886110 (+ 401us) server_negotiation.cc:407] Connection header received
0806 02:46:38.886351 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:38.886360 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:38.886426 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:38.886548 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:38.887419 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:38.888165 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:38.889485 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:38.889913 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:38.890064 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:38.896181 (+ 6117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.66:45849: BlockingRecv error: recv got EOF from 127.1.210.66:45849 (error 108)
Metrics: {"server-negotiator.queue_time_us":177}
Aug 06 02:46:39 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:39 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:39.381316 11257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:39.371678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.67:34255)
0806 02:46:39.372319 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:39.372344 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:39.372508 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:39.372863 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:39.372867 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:39.373319 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:39.373543 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:39.373555 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:39.374715 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:39.374720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:39.375690 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:39.375704 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:39.376501 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:39.376516 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:39.376719 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:39.377318 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:39.377334 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:39.380967 (+ 3633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":371,"threads_started":1}
W20260806 02:46:39.381587 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:39.371734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:34255 (local address 127.1.210.126:33657)
0806 02:46:39.371913 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:39.371918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:39.372424 (+ 506us) server_negotiation.cc:407] Connection header received
0806 02:46:39.372633 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:39.372637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:39.372706 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:39.372801 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:39.373717 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:39.374491 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:39.375859 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:39.376308 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:39.376463 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:39.381258 (+ 4795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.67:34255: BlockingRecv error: recv got EOF from 127.1.210.67:34255 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 06 02:46:39 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:39 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:39.708973 11258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:39.697566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.65:50561)
0806 02:46:39.698219 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:39.698247 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:39.698445 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:39.698863 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:39.698870 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:39.699472 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:39.699642 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:39.699652 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:39.700886 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:39.700898 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:39.701861 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:39.701876 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:39.702714 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:39.702721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:39.702907 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:39.703662 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:39.703681 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:39.708722 (+ 5041us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":356,"threads_started":1}
W20260806 02:46:39.709520 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:39.697566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:50561 (local address 127.1.210.126:33657)
0806 02:46:39.697860 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:39.697869 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:39.698373 (+ 504us) server_negotiation.cc:407] Connection header received
0806 02:46:39.698599 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:39.698603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:39.698670 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:39.698806 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:39.699801 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:39.700725 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:39.702031 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:39.702479 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:39.702632 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:39.709239 (+ 6607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.65:50561: BlockingRecv error: recv got EOF from 127.1.210.65:50561 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Aug 06 02:46:39 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:39 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:39.910872 11259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:39.898825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.66:49703)
0806 02:46:39.899561 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:39.899629 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:39.899867 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:39.900251 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:39.900256 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:39.900726 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:39.900925 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:39.900936 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:39.902232 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:39.902240 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:39.903609 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:39.903624 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:39.904529 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:39.904536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:39.904758 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:39.905527 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:39.905547 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:39.910492 (+ 4945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":527,"threads_started":1}
W20260806 02:46:39.911088 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:39.899054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:49703 (local address 127.1.210.126:33657)
0806 02:46:39.899369 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:39.899378 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:39.899790 (+ 412us) server_negotiation.cc:407] Connection header received
0806 02:46:39.900020 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:39.900026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:39.900099 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:39.900207 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:39.901103 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:39.902019 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:39.903808 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:39.904342 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:39.904519 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:39.910813 (+ 6294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.66:49703: BlockingRecv error: recv got EOF from 127.1.210.66:49703 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Aug 06 02:46:40 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:40 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.67@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:40.394398 11260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:40.382774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.67:56579)
0806 02:46:40.383439 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:40.383465 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:40.383646 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:40.384186 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:40.384191 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:40.384782 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:40.384984 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:40.384996 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:40.386159 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:40.386163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:40.387287 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:40.387301 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:40.388083 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:40.388088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:40.388317 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:40.389130 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:40.389149 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:40.394049 (+ 4900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":360,"threads_started":1}
W20260806 02:46:40.394660 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:40.382900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:56579 (local address 127.1.210.126:33657)
0806 02:46:40.383125 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:40.383130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:40.383599 (+ 469us) server_negotiation.cc:407] Connection header received
0806 02:46:40.383901 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:40.383905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:40.383982 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:40.384077 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:40.385192 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:40.385984 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:40.387450 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:40.387877 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:40.387994 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:40.394409 (+ 6415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.67:56579: BlockingRecv error: recv got EOF from 127.1.210.67:56579 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 06 02:46:40 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:40 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.65@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:40.722517 11261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:40.710748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.65:55593)
0806 02:46:40.711455 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:40.711475 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:40.711634 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:40.712087 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:40.712091 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:40.712579 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:40.712903 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:40.712914 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:40.714170 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:40.714183 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:40.715356 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:40.715373 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:40.716393 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:40.716400 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:40.716619 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:40.717508 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:40.717540 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:40.722135 (+ 4595us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":373,"threads_started":1}
W20260806 02:46:40.723161 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:40.711005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:55593 (local address 127.1.210.126:33657)
0806 02:46:40.711282 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:40.711288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:40.711588 (+ 300us) server_negotiation.cc:407] Connection header received
0806 02:46:40.711797 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:40.711802 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:40.711873 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:40.711974 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:40.713113 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:40.713971 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:40.715568 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:40.716153 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:40.716412 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:40.722606 (+ 6194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.65:55593: BlockingRecv error: recv got EOF from 127.1.210.65:55593 (error 108)
Metrics: {"server-negotiator.queue_time_us":137}
I20260806 02:46:40.914975 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 10795
Aug 06 02:46:40 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:40 dist-test-slave-8tdl krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.210.66@KRBTEST.COM for oryx/127.1.210.126@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:40.925515 11262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:40.913406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:33657 (local address 127.1.210.66:36181)
0806 02:46:40.913978 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:40.913995 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:40.914156 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:40.914630 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:40.914635 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:40.915377 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:40.915665 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:40.915683 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:40.917522 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:40.917529 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:40.918552 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:40.918567 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:40.919361 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:40.919368 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:40.919671 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:40.920717 (+ 1046us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:40.920738 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:40.924933 (+ 4195us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.210.126:33657: Server oryx/127.1.210.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":284,"threads_started":1}
W20260806 02:46:40.925649 11219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:40.913625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:36181 (local address 127.1.210.126:33657)
0806 02:46:40.913840 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:40.913846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:40.914130 (+ 284us) server_negotiation.cc:407] Connection header received
0806 02:46:40.914295 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:40.914300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:40.914376 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:40.914477 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:40.915890 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:40.917267 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:40.918752 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:40.919185 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:40.919339 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:40.925276 (+ 5937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.210.66:36181: BlockingRecv error: recv got EOF from 127.1.210.66:36181 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
I20260806 02:46:40.932686 11264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:40.916599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57900 (local address 127.1.210.65:37937)
0806 02:46:40.917672 (+ 1073us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:40.917679 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:40.917707 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:46:40.917769 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:40.917776 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:40.918023 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:40.918261 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:40.920537 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:40.921928 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:40.923644 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:40.924609 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:40.924781 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:40.928174 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:40.928185 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:40.928201 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:40.928252 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:40.931541 (+ 3289us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.210.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0806 02:46:40.932257 (+ 716us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57900: Request ticket server kudu/127.1.210.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":835,"thread_start_us":675,"threads_started":1}
W20260806 02:46:40.932711 1865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 10795: Not authorized: Client connection negotiation failed: client connection to 127.1.210.65:37937: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.210.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260806 02:46:40.932830 11264 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57900: Request ticket server kudu/127.1.210.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260806 02:46:40.932941 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 10795
I20260806 02:46:40.953534 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 10948
I20260806 02:46:40.971213 11265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:40.955007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49776 (local address 127.1.210.66:44759)
0806 02:46:40.955798 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:40.955807 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:40.955841 (+ 34us) server_negotiation.cc:407] Connection header received
0806 02:46:40.955933 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:40.955941 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:40.956304 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:40.956531 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:40.958529 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:40.960155 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:40.962147 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:40.962942 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:40.963080 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:40.966201 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:40.966214 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:40.966231 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:40.966281 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:40.970423 (+ 4142us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.210.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0806 02:46:40.970875 (+ 452us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49776: Request ticket server kudu/127.1.210.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":401,"threads_started":1}
W20260806 02:46:40.971371 11265 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49776: Request ticket server kudu/127.1.210.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260806 02:46:40.971483 1865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 10948: Not authorized: Client connection negotiation failed: client connection to 127.1.210.66:44759: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.210.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260806 02:46:40.971669 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 10948
I20260806 02:46:40.991333 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 11084
I20260806 02:46:41.014933 11266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:40.995660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35006 (local address 127.1.210.67:34335)
0806 02:46:40.997400 (+ 1740us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:40.997411 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:40.997447 (+ 36us) server_negotiation.cc:407] Connection header received
0806 02:46:40.997542 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:40.997566 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:40.997928 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:40.998932 (+ 1004us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:41.000754 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:41.002363 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:41.004230 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:41.005041 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:41.005216 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:41.010057 (+ 4841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:41.010070 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:41.010086 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:41.010143 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:41.013969 (+ 3826us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.210.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0806 02:46:41.014480 (+ 511us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35006: Request ticket server kudu/127.1.210.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":1442,"thread_start_us":1104,"threads_started":1}
W20260806 02:46:41.015091 11266 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35006: Request ticket server kudu/127.1.210.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260806 02:46:41.015108 1865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 11084: Not authorized: Client connection negotiation failed: client connection to 127.1.210.67:34335: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.210.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260806 02:46:41.015349 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 11084
I20260806 02:46:41.035472 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 10724
I20260806 02:46:41.037173 10788 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:46:41.180413 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 10724
2026-08-06T02:46:41Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21768 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-06T02:46:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:46:41Z Disabled control of system clock
I20260806 02:46:41.286571 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:39599
--webserver_interface=127.1.210.126
--webserver_port=0
--builtin_ntp_servers=127.1.210.84:38073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:39599
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 02:46:41.715188 11271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:41.715623 11271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:41.715713 11271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:41.728775 11271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:46:41.728982 11271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:41.729105 11271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:41.729225 11271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:46:41.729301 11271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:46:41.741014 11271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:38073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:39599
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:39599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:46:41.742525 11271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:41.745057 11271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:41.763370 11279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:41.763468 11277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:41.763311 11276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:41.768891 11271 server_base.cc:1034] running on GCE node
I20260806 02:46:41.770277 11271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:41.772472 11271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:41.773789 11271 hybrid_clock.cc:648] HybridClock initialized: now 1785984401773692 us; error 110 us; skew 500 ppm
I20260806 02:46:41.774376 11271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:46:41.783176 11271 webserver.cc:460] Webserver started at http://127.1.210.126:37017/ using document root <none> and password file <none>
I20260806 02:46:41.784293 11271 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:41.784420 11271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:41.785126 11271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:41.790387 11271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "fcae45ac466d45aaa722908368b984e9"
format_stamp: "Formatted at 2026-08-06 02:46:41 on dist-test-slave-8tdl"
server_key: "3b9c3a1e6875f4439178dff929b75314"
server_key_iv: "504a26bc10a58f7471776a9115dca828"
server_key_version: "encryptionkey@0"
I20260806 02:46:41.791739 11271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "fcae45ac466d45aaa722908368b984e9"
format_stamp: "Formatted at 2026-08-06 02:46:41 on dist-test-slave-8tdl"
server_key: "3b9c3a1e6875f4439178dff929b75314"
server_key_iv: "504a26bc10a58f7471776a9115dca828"
server_key_version: "encryptionkey@0"
I20260806 02:46:41.800149 11271 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.002s sys 0.005s
I20260806 02:46:41.805984 11285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:41.808212 11271 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260806 02:46:41.808451 11271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "fcae45ac466d45aaa722908368b984e9"
format_stamp: "Formatted at 2026-08-06 02:46:41 on dist-test-slave-8tdl"
server_key: "3b9c3a1e6875f4439178dff929b75314"
server_key_iv: "504a26bc10a58f7471776a9115dca828"
server_key_version: "encryptionkey@0"
I20260806 02:46:41.808717 11271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:41.833415 11271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:41.834853 11271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:41.835515 11271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:41.870251 11336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:39599 every 8 connection(s)
I20260806 02:46:41.870365 11271 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:39599
I20260806 02:46:41.872717 11271 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:46:41.878250 11337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:41.882912 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 11271
I20260806 02:46:41.883172 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:46:41.884225 1865 external_mini_cluster.cc:1428] Setting key 11b61034425fde69bb52f5d3039d793e
I20260806 02:46:41.898118 11337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcae45ac466d45aaa722908368b984e9: Bootstrap starting.
I20260806 02:46:41.903098 11337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcae45ac466d45aaa722908368b984e9: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:41.905244 11337 log.cc:824] T 00000000000000000000000000000000 P fcae45ac466d45aaa722908368b984e9: Log is configured to *not* fsync() on all Append() calls
I20260806 02:46:41.907394 11340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:41.888182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58732 (local address 127.1.210.126:39599)
0806 02:46:41.889605 (+ 1423us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:41.889620 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:41.889672 (+ 52us) server_negotiation.cc:407] Connection header received
0806 02:46:41.892443 (+ 2771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:41.892485 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:41.893337 (+ 852us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:41.894041 (+ 704us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:41.895190 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:41.897425 (+ 2235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:41.899353 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:41.900202 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:41.900465 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:41.902804 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:41.902816 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:41.902836 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:46:41.904360 (+ 1524us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:41.904580 (+ 220us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:46:41.904609 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:41.904794 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:41.905117 (+ 323us) server_negotiation.cc:299] Negotiation successful
0806 02:46:41.905563 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":869,"thread_start_us":508,"threads_started":1}
I20260806 02:46:41.909655 11337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcae45ac466d45aaa722908368b984e9: No bootstrap required, opened a new log
I20260806 02:46:41.921943 11337 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcae45ac466d45aaa722908368b984e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcae45ac466d45aaa722908368b984e9" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39599 } }
I20260806 02:46:41.922480 11337 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcae45ac466d45aaa722908368b984e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:41.922638 11337 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcae45ac466d45aaa722908368b984e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcae45ac466d45aaa722908368b984e9, State: Initialized, Role: FOLLOWER
I20260806 02:46:41.923538 11337 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcae45ac466d45aaa722908368b984e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcae45ac466d45aaa722908368b984e9" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39599 } }
I20260806 02:46:41.923862 11337 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcae45ac466d45aaa722908368b984e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:46:41.924064 11337 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcae45ac466d45aaa722908368b984e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:46:41.924507 11337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcae45ac466d45aaa722908368b984e9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:41.928835 11337 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcae45ac466d45aaa722908368b984e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcae45ac466d45aaa722908368b984e9" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39599 } }
I20260806 02:46:41.929612 11337 leader_election.cc:302] T 00000000000000000000000000000000 P fcae45ac466d45aaa722908368b984e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcae45ac466d45aaa722908368b984e9; no voters:
I20260806 02:46:41.930815 11337 leader_election.cc:288] T 00000000000000000000000000000000 P fcae45ac466d45aaa722908368b984e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:46:41.930928 11342 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcae45ac466d45aaa722908368b984e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:46:41.932287 11342 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcae45ac466d45aaa722908368b984e9 [term 1 LEADER]: Becoming Leader. State: Replica: fcae45ac466d45aaa722908368b984e9, State: Running, Role: LEADER
I20260806 02:46:41.933210 11342 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcae45ac466d45aaa722908368b984e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcae45ac466d45aaa722908368b984e9" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39599 } }
I20260806 02:46:41.933840 11337 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcae45ac466d45aaa722908368b984e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:46:41.945026 11344 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcae45ac466d45aaa722908368b984e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcae45ac466d45aaa722908368b984e9. Latest consensus state: current_term: 1 leader_uuid: "fcae45ac466d45aaa722908368b984e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcae45ac466d45aaa722908368b984e9" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39599 } } }
I20260806 02:46:41.944497 11343 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcae45ac466d45aaa722908368b984e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcae45ac466d45aaa722908368b984e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcae45ac466d45aaa722908368b984e9" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39599 } } }
I20260806 02:46:41.945566 11343 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcae45ac466d45aaa722908368b984e9 [sys.catalog]: This master's current role is: LEADER
I20260806 02:46:41.945605 11344 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcae45ac466d45aaa722908368b984e9 [sys.catalog]: This master's current role is: LEADER
I20260806 02:46:41.949296 11353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:46:41.956266 11353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:46:41.970057 11353 catalog_manager.cc:1347] Generated new cluster ID: 322a9c53b4eb4efca27a9bfa5848701c
I20260806 02:46:41.970218 11353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:46:42.000953 11353 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:46:42.003777 11353 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:46:42.016345 11353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcae45ac466d45aaa722908368b984e9: Generated new TSK 0
I20260806 02:46:42.018601 11353 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:46:42.040886 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.65:0
--local_ip_for_outbound_sockets=127.1.210.65
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:39599
--builtin_ntp_servers=127.1.210.84:38073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260806 02:46:42.511296 11362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:42.511670 11362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:42.511831 11362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:42.522330 11362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:42.522454 11362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:42.522655 11362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.65
I20260806 02:46:42.536880 11362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:38073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:39599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.65
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:46:42.538491 11362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:42.540551 11362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:42.558527 11367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:42.558331 11370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:42.558324 11368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:42.559508 11362 server_base.cc:1034] running on GCE node
I20260806 02:46:42.560441 11362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:42.561619 11362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:42.563062 11362 hybrid_clock.cc:648] HybridClock initialized: now 1785984402562911 us; error 97 us; skew 500 ppm
I20260806 02:46:42.563508 11362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:46:42.566111 11362 webserver.cc:460] Webserver started at http://127.1.210.65:39895/ using document root <none> and password file <none>
I20260806 02:46:42.566999 11362 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:42.567150 11362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:42.567533 11362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:42.570803 11362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/data/instance:
uuid: "b641e73f6e4d4d45b2b73aa354f93ab1"
format_stamp: "Formatted at 2026-08-06 02:46:42 on dist-test-slave-8tdl"
server_key: "828aa8eb2e2799ec5a2267e3fd9103dd"
server_key_iv: "375258564755157829829244a75f5a17"
server_key_version: "encryptionkey@0"
I20260806 02:46:42.572831 11362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance:
uuid: "b641e73f6e4d4d45b2b73aa354f93ab1"
format_stamp: "Formatted at 2026-08-06 02:46:42 on dist-test-slave-8tdl"
server_key: "828aa8eb2e2799ec5a2267e3fd9103dd"
server_key_iv: "375258564755157829829244a75f5a17"
server_key_version: "encryptionkey@0"
I20260806 02:46:42.582846 11362 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.011s sys 0.000s
I20260806 02:46:42.587604 11376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:42.589608 11362 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:46:42.589910 11362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/wal
uuid: "b641e73f6e4d4d45b2b73aa354f93ab1"
format_stamp: "Formatted at 2026-08-06 02:46:42 on dist-test-slave-8tdl"
server_key: "828aa8eb2e2799ec5a2267e3fd9103dd"
server_key_iv: "375258564755157829829244a75f5a17"
server_key_version: "encryptionkey@0"
I20260806 02:46:42.590276 11362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:42.616613 11362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:42.617664 11362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:42.618062 11362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:42.619658 11362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:46:42.622059 11362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:42.622215 11362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:42.622371 11362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:42.622484 11362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:42.667591 11362 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.65:40471
I20260806 02:46:42.667708 11489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.65:40471 every 8 connection(s)
I20260806 02:46:42.669301 11362 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
I20260806 02:46:42.674018 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 11362
I20260806 02:46:42.675136 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance
I20260806 02:46:42.677589 1865 external_mini_cluster.cc:1428] Setting key a8a082c1040db3c670084dc9d7bb29f7
I20260806 02:46:42.682111 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.66:0
--local_ip_for_outbound_sockets=127.1.210.66
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:39599
--builtin_ntp_servers=127.1.210.84:38073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 02:46:42.694074 11492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:42.672792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:53837 (local address 127.1.210.126:39599)
0806 02:46:42.673505 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:42.673515 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:42.678580 (+ 5065us) server_negotiation.cc:407] Connection header received
0806 02:46:42.681760 (+ 3180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:42.681769 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:42.681873 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:42.682015 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:42.684058 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:42.686306 (+ 2248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:42.689382 (+ 3076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:42.690287 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:42.690451 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:42.692823 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:42.692833 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:42.692842 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:46:42.692874 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:42.692940 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:46:42.692971 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:42.693116 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:42.693435 (+ 319us) server_negotiation.cc:299] Negotiation successful
0806 02:46:42.693567 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":360,"threads_started":1}
I20260806 02:46:42.695676 11493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:42.673547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:39599 (local address 127.1.210.65:53837)
0806 02:46:42.678290 (+ 4743us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:42.678349 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:42.681417 (+ 3068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:42.682457 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:42.682482 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:42.682630 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:42.683766 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:42.683800 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:42.686598 (+ 2798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:42.686605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:42.689126 (+ 2521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:42.689151 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:42.690665 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:42.690725 (+ 60us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:42.691262 (+ 537us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:42.692220 (+ 958us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:46:42.692223 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:42.692281 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:46:42.692459 (+ 178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:42.693159 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:42.693171 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:42.693188 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 02:46:42.693383 (+ 195us) client_negotiation.cc:240] Negotiation successful
0806 02:46:42.693797 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2832,"thread_start_us":1989,"threads_started":1}
I20260806 02:46:42.698011 11490 heartbeater.cc:347] Connected to a master server at 127.1.210.126:39599
I20260806 02:46:42.698446 11490 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:42.699733 11490 heartbeater.cc:511] Master 127.1.210.126:39599 requested a full tablet report, sending...
I20260806 02:46:42.705062 11302 ts_manager.cc:194] Registered new tserver with Master: b641e73f6e4d4d45b2b73aa354f93ab1 (127.1.210.65:40471)
I20260806 02:46:42.707722 11302 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.210.65:53837
W20260806 02:46:43.120890 11494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:43.121276 11494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:43.121444 11494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:43.130640 11494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:43.130803 11494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:43.131009 11494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.66
I20260806 02:46:43.141285 11494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:38073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:39599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.66
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:46:43.142797 11494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:43.144898 11494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:43.160753 11499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:43.160835 11502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:43.160869 11500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:43.161494 11494 server_base.cc:1034] running on GCE node
I20260806 02:46:43.162240 11494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:43.163501 11494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:43.164826 11494 hybrid_clock.cc:648] HybridClock initialized: now 1785984403164805 us; error 77 us; skew 500 ppm
I20260806 02:46:43.165282 11494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:46:43.168263 11494 webserver.cc:460] Webserver started at http://127.1.210.66:37857/ using document root <none> and password file <none>
I20260806 02:46:43.169174 11494 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:43.169354 11494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:43.169731 11494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:43.173488 11494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/data/instance:
uuid: "e064c5753d0a4dc388a228061951f225"
format_stamp: "Formatted at 2026-08-06 02:46:43 on dist-test-slave-8tdl"
server_key: "116b25301ee41a8be4df7611cdbe74d4"
server_key_iv: "1a1f05b993708a2dc6e2b91997a3de62"
server_key_version: "encryptionkey@0"
I20260806 02:46:43.174605 11494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance:
uuid: "e064c5753d0a4dc388a228061951f225"
format_stamp: "Formatted at 2026-08-06 02:46:43 on dist-test-slave-8tdl"
server_key: "116b25301ee41a8be4df7611cdbe74d4"
server_key_iv: "1a1f05b993708a2dc6e2b91997a3de62"
server_key_version: "encryptionkey@0"
I20260806 02:46:43.181535 11494 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260806 02:46:43.186022 11508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:43.187618 11494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 02:46:43.187835 11494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/wal
uuid: "e064c5753d0a4dc388a228061951f225"
format_stamp: "Formatted at 2026-08-06 02:46:43 on dist-test-slave-8tdl"
server_key: "116b25301ee41a8be4df7611cdbe74d4"
server_key_iv: "1a1f05b993708a2dc6e2b91997a3de62"
server_key_version: "encryptionkey@0"
I20260806 02:46:43.188100 11494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:43.223843 11494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:43.224887 11494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:43.225256 11494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:43.226416 11494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:46:43.228579 11494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:43.228713 11494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:43.228870 11494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:43.228981 11494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:43.269652 11494 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.66:35243
I20260806 02:46:43.269750 11621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.66:35243 every 8 connection(s)
I20260806 02:46:43.271245 11494 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
I20260806 02:46:43.276510 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 11494
I20260806 02:46:43.276710 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance
I20260806 02:46:43.277410 1865 external_mini_cluster.cc:1428] Setting key 3b410f1a34ce30a1cef55c3be7945efe
I20260806 02:46:43.280997 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.67:0
--local_ip_for_outbound_sockets=127.1.210.67
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:39599
--builtin_ntp_servers=127.1.210.84:38073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 02:46:43.288813 11624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:43.274396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:35655 (local address 127.1.210.126:39599)
0806 02:46:43.274952 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:43.274957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:43.276356 (+ 1399us) server_negotiation.cc:407] Connection header received
0806 02:46:43.279068 (+ 2712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:43.279074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:43.279150 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:43.279282 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:43.280820 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:43.281776 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:43.285188 (+ 3412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:43.285649 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:43.285777 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:43.287822 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:43.287828 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:43.287831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:46:43.287861 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:43.287907 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:46:43.287920 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:43.287996 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:43.288405 (+ 409us) server_negotiation.cc:299] Negotiation successful
0806 02:46:43.288519 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":324,"threads_started":1}
I20260806 02:46:43.290534 11625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:43.275077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:39599 (local address 127.1.210.66:35655)
0806 02:46:43.276144 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:43.276210 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:43.278788 (+ 2578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:43.279547 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:43.279564 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:43.279700 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:43.280604 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:43.280631 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:43.281950 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:43.281959 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:43.284954 (+ 2995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:43.284974 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:43.285930 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:43.285940 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:43.286355 (+ 415us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:43.287352 (+ 997us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:46:43.287359 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:43.287420 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:46:43.287563 (+ 143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:43.288030 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:43.288044 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:43.288066 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:46:43.288384 (+ 318us) client_negotiation.cc:240] Negotiation successful
0806 02:46:43.288941 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":314,"threads_started":1}
I20260806 02:46:43.292616 11622 heartbeater.cc:347] Connected to a master server at 127.1.210.126:39599
I20260806 02:46:43.293048 11622 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:43.294024 11622 heartbeater.cc:511] Master 127.1.210.126:39599 requested a full tablet report, sending...
I20260806 02:46:43.295950 11302 ts_manager.cc:194] Registered new tserver with Master: e064c5753d0a4dc388a228061951f225 (127.1.210.66:35243)
I20260806 02:46:43.297353 11302 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.210.66:35655
W20260806 02:46:43.665007 11626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:43.665347 11626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:43.665509 11626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:43.673785 11626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:43.673893 11626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:43.674002 11626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.67
I20260806 02:46:43.684088 11626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:38073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:39599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.67
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:46:43.685532 11626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:43.687554 11626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:43.702392 11631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:43.702512 11634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:43.702441 11632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:43.703764 11626 server_base.cc:1034] running on GCE node
I20260806 02:46:43.704463 11626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:43.705621 11626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:43.706918 11626 hybrid_clock.cc:648] HybridClock initialized: now 1785984403706829 us; error 77 us; skew 500 ppm
I20260806 02:46:43.707295 11626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:46:43.710255 11626 webserver.cc:460] Webserver started at http://127.1.210.67:37341/ using document root <none> and password file <none>
I20260806 02:46:43.711135 11626 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:43.711237 11626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:43.711874 11626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:43.713063 11490 heartbeater.cc:503] Master 127.1.210.126:39599 was elected leader, sending a full tablet report...
I20260806 02:46:43.715793 11626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/data/instance:
uuid: "ec1b3e3bbd3740d784984f46e0180be3"
format_stamp: "Formatted at 2026-08-06 02:46:43 on dist-test-slave-8tdl"
server_key: "869db8fb031f1696099697b5533cf067"
server_key_iv: "fda9d3f07f00bde05d0e916c88b5768a"
server_key_version: "encryptionkey@0"
I20260806 02:46:43.717337 11626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance:
uuid: "ec1b3e3bbd3740d784984f46e0180be3"
format_stamp: "Formatted at 2026-08-06 02:46:43 on dist-test-slave-8tdl"
server_key: "869db8fb031f1696099697b5533cf067"
server_key_iv: "fda9d3f07f00bde05d0e916c88b5768a"
server_key_version: "encryptionkey@0"
I20260806 02:46:43.726439 11626 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.001s
I20260806 02:46:43.731669 11640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:43.733232 11626 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260806 02:46:43.733443 11626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/wal
uuid: "ec1b3e3bbd3740d784984f46e0180be3"
format_stamp: "Formatted at 2026-08-06 02:46:43 on dist-test-slave-8tdl"
server_key: "869db8fb031f1696099697b5533cf067"
server_key_iv: "fda9d3f07f00bde05d0e916c88b5768a"
server_key_version: "encryptionkey@0"
I20260806 02:46:43.733745 11626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:43.755152 11626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:43.756201 11626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:43.756511 11626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:43.757622 11626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:46:43.759338 11626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:43.759443 11626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:43.759528 11626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:43.759588 11626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:43.803274 11626 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.67:33903
I20260806 02:46:43.803398 11753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.67:33903 every 8 connection(s)
I20260806 02:46:43.804728 11626 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
I20260806 02:46:43.805863 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 11626
I20260806 02:46:43.806039 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance
I20260806 02:46:43.806541 1865 external_mini_cluster.cc:1428] Setting key acb792d129353cbc23bcbd9f7916da4d
I20260806 02:46:43.820122 11756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:43.807963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:60901 (local address 127.1.210.126:39599)
0806 02:46:43.808566 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:43.808574 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:43.810349 (+ 1775us) server_negotiation.cc:407] Connection header received
0806 02:46:43.812316 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:43.812321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:43.812402 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:43.812532 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:43.813777 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:43.814828 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:43.816903 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:43.817402 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:43.817526 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:43.819196 (+ 1670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:43.819201 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:43.819205 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:46:43.819226 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:43.819293 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:46:43.819308 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:43.819387 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:43.819686 (+ 299us) server_negotiation.cc:299] Negotiation successful
0806 02:46:43.819773 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":240,"threads_started":1}
I20260806 02:46:43.822317 11757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:43.808910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:39599 (local address 127.1.210.67:60901)
0806 02:46:43.810170 (+ 1260us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:43.810223 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:43.812008 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:43.812761 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:43.812769 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:43.812872 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:43.813606 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:43.813625 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:43.815030 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:43.815034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:43.816741 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:43.816754 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:43.817654 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:43.817659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:43.817911 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:43.818800 (+ 889us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:46:43.818802 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:43.818857 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:46:43.818966 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:43.819453 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:43.819462 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:43.819469 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 02:46:43.819673 (+ 204us) client_negotiation.cc:240] Negotiation successful
0806 02:46:43.820079 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":348,"threads_started":1}
I20260806 02:46:43.824765 11754 heartbeater.cc:347] Connected to a master server at 127.1.210.126:39599
I20260806 02:46:43.825208 11754 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:43.826093 11754 heartbeater.cc:511] Master 127.1.210.126:39599 requested a full tablet report, sending...
I20260806 02:46:43.828029 11301 ts_manager.cc:194] Registered new tserver with Master: ec1b3e3bbd3740d784984f46e0180be3 (127.1.210.67:33903)
I20260806 02:46:43.828953 11301 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.210.67:60901
I20260806 02:46:43.837728 1865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 02:46:43.846040 11756 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:43.843729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58734 (local address 127.1.210.126:39599)
0806 02:46:43.843958 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:43.843963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:43.844736 (+ 773us) server_negotiation.cc:407] Connection header received
0806 02:46:43.844986 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:43.844991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:43.845160 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:43.845301 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:43.845732 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58734: BlockingRecv error: recv got EOF from 127.0.0.1:58734 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 02:46:43.847186 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 11362
I20260806 02:46:43.857689 11764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:43.848254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47092 (local address 127.1.210.65:40471)
0806 02:46:43.848904 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:43.848912 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:43.848948 (+ 36us) server_negotiation.cc:407] Connection header received
0806 02:46:43.849021 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:43.849028 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:43.849268 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:43.849421 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:43.849882 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:43.851340 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:43.852761 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:43.853474 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:43.853606 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:43.855162 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:43.855172 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:43.855187 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:46:43.856558 (+ 1371us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:43.856621 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:46:43.856646 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:43.856776 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:43.856954 (+ 178us) server_negotiation.cc:299] Negotiation successful
0806 02:46:43.857166 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":381,"threads_started":1}
I20260806 02:46:43.859375 11485 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:46:43.961714 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 11362
I20260806 02:46:43.982055 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 11494
I20260806 02:46:43.991914 11766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:43.983235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51402 (local address 127.1.210.66:35243)
0806 02:46:43.983956 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:43.983964 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:43.984004 (+ 40us) server_negotiation.cc:407] Connection header received
0806 02:46:43.984091 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:43.984099 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:43.984331 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:43.984504 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:43.984977 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:43.986124 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:43.988021 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:43.988667 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:43.988798 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:43.990065 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:43.990082 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:43.990102 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:46:43.990907 (+ 805us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:43.990963 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:46:43.991001 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:43.991106 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:43.991316 (+ 210us) server_negotiation.cc:299] Negotiation successful
0806 02:46:43.991452 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":382,"threads_started":1}
I20260806 02:46:43.993531 11617 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:46:44.089555 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 11494
I20260806 02:46:44.108374 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 11626
I20260806 02:46:44.118413 11768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:44.109597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55312 (local address 127.1.210.67:33903)
0806 02:46:44.110279 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:44.110286 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:44.110318 (+ 32us) server_negotiation.cc:407] Connection header received
0806 02:46:44.110379 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:44.110386 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:44.110723 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:44.110869 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:44.111373 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:44.112595 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:44.114272 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:44.115177 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:44.115306 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:44.116667 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:44.116678 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:44.116692 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:46:44.117507 (+ 815us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:44.117560 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:46:44.117582 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:44.117683 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:44.117904 (+ 221us) server_negotiation.cc:299] Negotiation successful
0806 02:46:44.118088 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":347,"threads_started":1}
I20260806 02:46:44.119908 11749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:46:44.210114 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 11626
I20260806 02:46:44.229305 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 11271
I20260806 02:46:44.231057 11332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:46:44.347437 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 11271
2026-08-06T02:46:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (3155 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-06T02:46:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:46:44Z Disabled control of system clock
I20260806 02:46:44.432592 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:34983
--webserver_interface=127.1.210.126
--webserver_port=0
--builtin_ntp_servers=127.1.210.84:34467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:34983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 02:46:44.794637 11774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:44.795140 11774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:44.795322 11774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:44.803073 11774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:46:44.803200 11774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:44.803270 11774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:44.803381 11774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:46:44.803447 11774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:46:44.813392 11774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:34467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:34983
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:34983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:46:44.814949 11774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:44.816915 11774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:44.830400 11782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:44.830585 11780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:44.830739 11779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:44.831626 11774 server_base.cc:1034] running on GCE node
I20260806 02:46:44.832661 11774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:44.834298 11774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:44.835517 11774 hybrid_clock.cc:648] HybridClock initialized: now 1785984404835437 us; error 77 us; skew 500 ppm
I20260806 02:46:44.835907 11774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:46:44.838619 11774 webserver.cc:460] Webserver started at http://127.1.210.126:42315/ using document root <none> and password file <none>
I20260806 02:46:44.839468 11774 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:44.839592 11774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:44.839953 11774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:44.843258 11774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "db4a882798ce431bad7d7699373fdaee"
format_stamp: "Formatted at 2026-08-06 02:46:44 on dist-test-slave-8tdl"
server_key: "f0cb50a097244bff90c7283145c1bff9"
server_key_iv: "fdde812cadd868995ddab8807c5331b8"
server_key_version: "encryptionkey@0"
I20260806 02:46:44.844192 11774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "db4a882798ce431bad7d7699373fdaee"
format_stamp: "Formatted at 2026-08-06 02:46:44 on dist-test-slave-8tdl"
server_key: "f0cb50a097244bff90c7283145c1bff9"
server_key_iv: "fdde812cadd868995ddab8807c5331b8"
server_key_version: "encryptionkey@0"
I20260806 02:46:44.850019 11774 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260806 02:46:44.853992 11788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:44.855561 11774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:46:44.855772 11774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "db4a882798ce431bad7d7699373fdaee"
format_stamp: "Formatted at 2026-08-06 02:46:44 on dist-test-slave-8tdl"
server_key: "f0cb50a097244bff90c7283145c1bff9"
server_key_iv: "fdde812cadd868995ddab8807c5331b8"
server_key_version: "encryptionkey@0"
I20260806 02:46:44.856026 11774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:44.862897 11774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:44.863860 11774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:44.864217 11774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:44.884963 11774 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:34983
I20260806 02:46:44.884994 11839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:34983 every 8 connection(s)
I20260806 02:46:44.886652 11774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:46:44.890506 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 11774
I20260806 02:46:44.890826 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:46:44.891041 11840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:44.891623 1865 external_mini_cluster.cc:1428] Setting key dae17a8abd0e61d5baed021b6feb95d3
I20260806 02:46:44.901036 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:44.893939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50450 (local address 127.1.210.126:34983)
0806 02:46:44.895131 (+ 1192us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:44.895143 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:44.895191 (+ 48us) server_negotiation.cc:407] Connection header received
0806 02:46:44.897290 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:44.897316 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:44.897853 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:44.898216 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:44.899406 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:44.899412 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:44.899425 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:46:44.899438 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:44.899499 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:46:44.899514 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:44.899691 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:44.899788 (+ 97us) server_negotiation.cc:299] Negotiation successful
0806 02:46:44.900082 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":423,"threads_started":1}
I20260806 02:46:44.905114 11840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db4a882798ce431bad7d7699373fdaee: Bootstrap starting.
I20260806 02:46:44.908456 11840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db4a882798ce431bad7d7699373fdaee: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:44.909552 11840 log.cc:824] T 00000000000000000000000000000000 P db4a882798ce431bad7d7699373fdaee: Log is configured to *not* fsync() on all Append() calls
I20260806 02:46:44.912719 11840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db4a882798ce431bad7d7699373fdaee: No bootstrap required, opened a new log
I20260806 02:46:44.920661 11840 raft_consensus.cc:348] T 00000000000000000000000000000000 P db4a882798ce431bad7d7699373fdaee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db4a882798ce431bad7d7699373fdaee" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 34983 } }
I20260806 02:46:44.921151 11840 raft_consensus.cc:374] T 00000000000000000000000000000000 P db4a882798ce431bad7d7699373fdaee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:44.921268 11840 raft_consensus.cc:729] T 00000000000000000000000000000000 P db4a882798ce431bad7d7699373fdaee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db4a882798ce431bad7d7699373fdaee, State: Initialized, Role: FOLLOWER
I20260806 02:46:44.921906 11840 consensus_queue.cc:260] T 00000000000000000000000000000000 P db4a882798ce431bad7d7699373fdaee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db4a882798ce431bad7d7699373fdaee" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 34983 } }
I20260806 02:46:44.922183 11840 raft_consensus.cc:388] T 00000000000000000000000000000000 P db4a882798ce431bad7d7699373fdaee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:46:44.922343 11840 raft_consensus.cc:482] T 00000000000000000000000000000000 P db4a882798ce431bad7d7699373fdaee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:46:44.922544 11840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db4a882798ce431bad7d7699373fdaee [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:44.924942 11840 raft_consensus.cc:504] T 00000000000000000000000000000000 P db4a882798ce431bad7d7699373fdaee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db4a882798ce431bad7d7699373fdaee" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 34983 } }
I20260806 02:46:44.925446 11840 leader_election.cc:302] T 00000000000000000000000000000000 P db4a882798ce431bad7d7699373fdaee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db4a882798ce431bad7d7699373fdaee; no voters:
I20260806 02:46:44.926083 11840 leader_election.cc:288] T 00000000000000000000000000000000 P db4a882798ce431bad7d7699373fdaee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:46:44.926257 11845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db4a882798ce431bad7d7699373fdaee [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:46:44.927103 11845 raft_consensus.cc:686] T 00000000000000000000000000000000 P db4a882798ce431bad7d7699373fdaee [term 1 LEADER]: Becoming Leader. State: Replica: db4a882798ce431bad7d7699373fdaee, State: Running, Role: LEADER
I20260806 02:46:44.927690 11845 consensus_queue.cc:237] T 00000000000000000000000000000000 P db4a882798ce431bad7d7699373fdaee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db4a882798ce431bad7d7699373fdaee" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 34983 } }
I20260806 02:46:44.928730 11840 sys_catalog.cc:564] T 00000000000000000000000000000000 P db4a882798ce431bad7d7699373fdaee [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:46:44.931917 11846 sys_catalog.cc:455] T 00000000000000000000000000000000 P db4a882798ce431bad7d7699373fdaee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db4a882798ce431bad7d7699373fdaee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db4a882798ce431bad7d7699373fdaee" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 34983 } } }
I20260806 02:46:44.932372 11846 sys_catalog.cc:458] T 00000000000000000000000000000000 P db4a882798ce431bad7d7699373fdaee [sys.catalog]: This master's current role is: LEADER
I20260806 02:46:44.932019 11847 sys_catalog.cc:455] T 00000000000000000000000000000000 P db4a882798ce431bad7d7699373fdaee [sys.catalog]: SysCatalogTable state changed. Reason: New leader db4a882798ce431bad7d7699373fdaee. Latest consensus state: current_term: 1 leader_uuid: "db4a882798ce431bad7d7699373fdaee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db4a882798ce431bad7d7699373fdaee" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 34983 } } }
I20260806 02:46:44.932783 11847 sys_catalog.cc:458] T 00000000000000000000000000000000 P db4a882798ce431bad7d7699373fdaee [sys.catalog]: This master's current role is: LEADER
I20260806 02:46:44.934175 11853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:46:44.939987 11853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:46:44.949095 11853 catalog_manager.cc:1347] Generated new cluster ID: efaa5f8e586c41ccab07e9d1496f1a8f
I20260806 02:46:44.949236 11853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:46:44.980436 11853 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:46:44.981045 11853 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:46:44.995715 11853 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db4a882798ce431bad7d7699373fdaee: Generated new TSK 0
I20260806 02:46:44.996690 11853 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:46:45.015118 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.65:0
--local_ip_for_outbound_sockets=127.1.210.65
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:34983
--builtin_ntp_servers=127.1.210.84:34467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 02:46:45.385365 11864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:45.385661 11864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:45.385814 11864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:45.393292 11864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:45.393410 11864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:45.393622 11864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.65
I20260806 02:46:45.403640 11864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:34467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:34983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.65
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:46:45.405009 11864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:45.407016 11864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:45.421528 11869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:45.421656 11870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:45.421656 11872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:45.422672 11864 server_base.cc:1034] running on GCE node
I20260806 02:46:45.423389 11864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:45.424456 11864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:45.425715 11864 hybrid_clock.cc:648] HybridClock initialized: now 1785984405425654 us; error 58 us; skew 500 ppm
I20260806 02:46:45.426080 11864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:46:45.429060 11864 webserver.cc:460] Webserver started at http://127.1.210.65:38473/ using document root <none> and password file <none>
I20260806 02:46:45.429895 11864 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:45.430030 11864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:45.430377 11864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:45.434115 11864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/data/instance:
uuid: "104dad7d5b994a6580b1e700dc3487cb"
format_stamp: "Formatted at 2026-08-06 02:46:45 on dist-test-slave-8tdl"
server_key: "a787967fefbc4d273795cd654075df02"
server_key_iv: "317afac96e8b3b9e72fd2aff710bbcc8"
server_key_version: "encryptionkey@0"
I20260806 02:46:45.435057 11864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance:
uuid: "104dad7d5b994a6580b1e700dc3487cb"
format_stamp: "Formatted at 2026-08-06 02:46:45 on dist-test-slave-8tdl"
server_key: "a787967fefbc4d273795cd654075df02"
server_key_iv: "317afac96e8b3b9e72fd2aff710bbcc8"
server_key_version: "encryptionkey@0"
I20260806 02:46:45.440482 11864 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 02:46:45.444259 11878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:45.445720 11864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:46:45.445864 11864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/wal
uuid: "104dad7d5b994a6580b1e700dc3487cb"
format_stamp: "Formatted at 2026-08-06 02:46:45 on dist-test-slave-8tdl"
server_key: "a787967fefbc4d273795cd654075df02"
server_key_iv: "317afac96e8b3b9e72fd2aff710bbcc8"
server_key_version: "encryptionkey@0"
I20260806 02:46:45.446059 11864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:45.452033 11864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:45.452826 11864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:45.453066 11864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:45.453949 11864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:46:45.455418 11864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:45.455508 11864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:45.455674 11864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:45.455775 11864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:45.491988 11864 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.65:34853
I20260806 02:46:45.492110 11990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.65:34853 every 8 connection(s)
I20260806 02:46:45.493448 11864 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
I20260806 02:46:45.496798 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 11864
I20260806 02:46:45.497025 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance
I20260806 02:46:45.497905 1865 external_mini_cluster.cc:1428] Setting key 8dadbc55c596670d1dbfe74f6a5ff528
I20260806 02:46:45.501178 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.66:0
--local_ip_for_outbound_sockets=127.1.210.66
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:34983
--builtin_ntp_servers=127.1.210.84:34467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 02:46:45.503868 11993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:45.496408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:49595 (local address 127.1.210.126:34983)
0806 02:46:45.496963 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:45.496972 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:45.498766 (+ 1794us) server_negotiation.cc:407] Connection header received
0806 02:46:45.501146 (+ 2380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:45.501155 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:45.501229 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:45.501360 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:45.502972 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:45.502981 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:45.502988 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:46:45.503013 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:45.503071 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:46:45.503089 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:45.503231 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:45.503428 (+ 197us) server_negotiation.cc:299] Negotiation successful
0806 02:46:45.503497 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":289,"threads_started":1}
I20260806 02:46:45.504865 11994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:45.497256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:34983 (local address 127.1.210.65:49595)
0806 02:46:45.498512 (+ 1256us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:45.498578 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:45.500835 (+ 2257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:45.501666 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:45.501676 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:45.501783 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:45.502581 (+ 798us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:46:45.502584 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:45.502627 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:46:45.502790 (+ 163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:45.503211 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:45.503220 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:45.503227 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 02:46:45.503471 (+ 244us) client_negotiation.cc:240] Negotiation successful
0806 02:46:45.503701 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":339,"threads_started":1}
I20260806 02:46:45.506706 11991 heartbeater.cc:347] Connected to a master server at 127.1.210.126:34983
I20260806 02:46:45.507099 11991 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:45.507763 11991 heartbeater.cc:511] Master 127.1.210.126:34983 requested a full tablet report, sending...
I20260806 02:46:45.510095 11805 ts_manager.cc:194] Registered new tserver with Master: 104dad7d5b994a6580b1e700dc3487cb (127.1.210.65:34853)
W20260806 02:46:45.860249 11995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:45.860599 11995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:45.860790 11995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:45.868630 11995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:45.868810 11995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:45.869019 11995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.66
I20260806 02:46:45.878801 11995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:34467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:34983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.66
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:46:45.880223 11995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:45.882010 11995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:45.898628 12001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:45.898880 12000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:45.899281 12003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:45.899729 11995 server_base.cc:1034] running on GCE node
I20260806 02:46:45.900415 11995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:45.901549 11995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:45.902793 11995 hybrid_clock.cc:648] HybridClock initialized: now 1785984405902730 us; error 56 us; skew 500 ppm
I20260806 02:46:45.903157 11995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:46:45.905786 11995 webserver.cc:460] Webserver started at http://127.1.210.66:35721/ using document root <none> and password file <none>
I20260806 02:46:45.906558 11995 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:45.906741 11995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:45.907102 11995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:45.910259 11995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/data/instance:
uuid: "867d0563896d4e519d5e290c029cffd9"
format_stamp: "Formatted at 2026-08-06 02:46:45 on dist-test-slave-8tdl"
server_key: "3f5186f2239d5b2a72a066f9c88b04bc"
server_key_iv: "68dd0d7dcea5b0d0e780559576c98104"
server_key_version: "encryptionkey@0"
I20260806 02:46:45.911147 11995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance:
uuid: "867d0563896d4e519d5e290c029cffd9"
format_stamp: "Formatted at 2026-08-06 02:46:45 on dist-test-slave-8tdl"
server_key: "3f5186f2239d5b2a72a066f9c88b04bc"
server_key_iv: "68dd0d7dcea5b0d0e780559576c98104"
server_key_version: "encryptionkey@0"
I20260806 02:46:45.917358 11995 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260806 02:46:45.921716 12009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:45.923349 11995 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:46:45.923553 11995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/wal
uuid: "867d0563896d4e519d5e290c029cffd9"
format_stamp: "Formatted at 2026-08-06 02:46:45 on dist-test-slave-8tdl"
server_key: "3f5186f2239d5b2a72a066f9c88b04bc"
server_key_iv: "68dd0d7dcea5b0d0e780559576c98104"
server_key_version: "encryptionkey@0"
I20260806 02:46:45.923820 11995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:45.930884 11995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:45.931799 11995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:45.932102 11995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:45.933198 11995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:46:45.934964 11995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:45.935096 11995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:45.935221 11995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:45.935307 11995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:45.972136 11995 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.66:39883
I20260806 02:46:45.972240 12122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.66:39883 every 8 connection(s)
I20260806 02:46:45.973572 11995 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
I20260806 02:46:45.981302 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 11995
I20260806 02:46:45.981499 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance
I20260806 02:46:45.982148 1865 external_mini_cluster.cc:1428] Setting key 157bacd809b77100588a4cd3e2a12e96
I20260806 02:46:45.982538 11993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:45.976443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:59149 (local address 127.1.210.126:34983)
0806 02:46:45.976668 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:45.976673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:45.978127 (+ 1454us) server_negotiation.cc:407] Connection header received
0806 02:46:45.979917 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:45.979922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:45.979976 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:45.980071 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:45.981750 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:45.981757 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:45.981762 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:46:45.981785 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:45.981831 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:46:45.981849 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:45.981997 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:45.982169 (+ 172us) server_negotiation.cc:299] Negotiation successful
0806 02:46:45.982245 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260806 02:46:45.983686 12125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:45.977008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:34983 (local address 127.1.210.66:59149)
0806 02:46:45.977953 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:45.977990 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:45.979650 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:45.980363 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:45.980373 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:45.980489 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:45.981335 (+ 846us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:46:45.981337 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:45.981405 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:46:45.981532 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:45.981957 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:45.981965 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:45.981971 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 02:46:45.982225 (+ 254us) client_negotiation.cc:240] Negotiation successful
0806 02:46:45.982480 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":286,"threads_started":1}
I20260806 02:46:45.985118 12123 heartbeater.cc:347] Connected to a master server at 127.1.210.126:34983
I20260806 02:46:45.985389 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.67:0
--local_ip_for_outbound_sockets=127.1.210.67
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:34983
--builtin_ntp_servers=127.1.210.84:34467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 02:46:45.985430 12123 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:45.986064 12123 heartbeater.cc:511] Master 127.1.210.126:34983 requested a full tablet report, sending...
I20260806 02:46:45.987643 11805 ts_manager.cc:194] Registered new tserver with Master: 867d0563896d4e519d5e290c029cffd9 (127.1.210.66:39883)
W20260806 02:46:46.344987 12126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:46.345297 12126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:46.345391 12126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:46.353052 12126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:46.353163 12126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:46.353276 12126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.67
I20260806 02:46:46.362766 12126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:34467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:34983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.67
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:46:46.364248 12126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:46.366119 12126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:46.380874 12131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:46.381006 12132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:46.380940 12134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:46.382226 12126 server_base.cc:1034] running on GCE node
I20260806 02:46:46.383034 12126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:46.384155 12126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:46.385428 12126 hybrid_clock.cc:648] HybridClock initialized: now 1785984406385378 us; error 71 us; skew 500 ppm
I20260806 02:46:46.385794 12126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:46:46.388393 12126 webserver.cc:460] Webserver started at http://127.1.210.67:45571/ using document root <none> and password file <none>
I20260806 02:46:46.389169 12126 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:46.389315 12126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:46.389672 12126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:46.392903 12126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/data/instance:
uuid: "9de7dd78409a4be8a7e345febb65f4ba"
format_stamp: "Formatted at 2026-08-06 02:46:46 on dist-test-slave-8tdl"
server_key: "c72318e509d73d0aced2e4e9c88858ef"
server_key_iv: "bbad2804a7b428756af7d9077ddf40be"
server_key_version: "encryptionkey@0"
I20260806 02:46:46.393890 12126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance:
uuid: "9de7dd78409a4be8a7e345febb65f4ba"
format_stamp: "Formatted at 2026-08-06 02:46:46 on dist-test-slave-8tdl"
server_key: "c72318e509d73d0aced2e4e9c88858ef"
server_key_iv: "bbad2804a7b428756af7d9077ddf40be"
server_key_version: "encryptionkey@0"
I20260806 02:46:46.399864 12126 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260806 02:46:46.403924 12140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:46.405416 12126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:46:46.405617 12126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/wal
uuid: "9de7dd78409a4be8a7e345febb65f4ba"
format_stamp: "Formatted at 2026-08-06 02:46:46 on dist-test-slave-8tdl"
server_key: "c72318e509d73d0aced2e4e9c88858ef"
server_key_iv: "bbad2804a7b428756af7d9077ddf40be"
server_key_version: "encryptionkey@0"
I20260806 02:46:46.405864 12126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:46.412096 12126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:46.413251 12126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:46.413596 12126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:46.414811 12126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:46:46.416495 12126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:46.416603 12126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:46.416697 12126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:46.416812 12126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:46.456014 12126 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.67:38563
I20260806 02:46:46.456084 12253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.67:38563 every 8 connection(s)
I20260806 02:46:46.457571 12126 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
I20260806 02:46:46.464795 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 12126
I20260806 02:46:46.465024 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance
I20260806 02:46:46.465631 1865 external_mini_cluster.cc:1428] Setting key ed0932cf23fd1720e4f8cec3e2a272c5
I20260806 02:46:46.467502 11993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:46.460679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:41755 (local address 127.1.210.126:34983)
0806 02:46:46.460902 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:46.460908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:46.462476 (+ 1568us) server_negotiation.cc:407] Connection header received
0806 02:46:46.464478 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:46.464483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:46.464550 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:46.464671 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:46.466386 (+ 1715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:46.466392 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:46.466397 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:46:46.466426 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:46.466493 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:46:46.466514 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:46.466654 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:46.467089 (+ 435us) server_negotiation.cc:299] Negotiation successful
0806 02:46:46.467161 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 02:46:46.468900 12256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:46.461284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:34983 (local address 127.1.210.67:41755)
0806 02:46:46.462295 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:46.462334 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:46.464116 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:46.464971 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:46.464986 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:46.465155 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:46.465972 (+ 817us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:46:46.465976 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:46.466036 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:46:46.466170 (+ 134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:46.466660 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:46.466668 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:46.466741 (+ 73us) client_negotiation.cc:769] Sending connection context
0806 02:46:46.467215 (+ 474us) client_negotiation.cc:240] Negotiation successful
0806 02:46:46.467514 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":318,"threads_started":1}
I20260806 02:46:46.470474 12254 heartbeater.cc:347] Connected to a master server at 127.1.210.126:34983
I20260806 02:46:46.470855 12254 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:46.471585 12254 heartbeater.cc:511] Master 127.1.210.126:34983 requested a full tablet report, sending...
I20260806 02:46:46.473284 11805 ts_manager.cc:194] Registered new tserver with Master: 9de7dd78409a4be8a7e345febb65f4ba (127.1.210.67:38563)
I20260806 02:46:46.482394 1865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 02:46:46.490372 11993 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:46.487928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50460 (local address 127.1.210.126:34983)
0806 02:46:46.488160 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:46.488165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:46.488848 (+ 683us) server_negotiation.cc:407] Connection header received
0806 02:46:46.489234 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:46.489238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:46.489298 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:46.489393 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:46.489926 (+ 533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50460: BlockingRecv error: recv got EOF from 127.0.0.1:50460 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 02:46:46.490754 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 11864
I20260806 02:46:46.496004 12263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:46.492356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54088 (local address 127.1.210.65:34853)
0806 02:46:46.493116 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:46.493123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:46.493150 (+ 27us) server_negotiation.cc:407] Connection header received
0806 02:46:46.493221 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:46.493227 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:46.493447 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:46.493643 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:46.495031 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:46.495037 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:46.495049 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:46:46.495063 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:46.495108 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:46:46.495126 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:46.495374 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:46.495459 (+ 85us) server_negotiation.cc:299] Negotiation successful
0806 02:46:46.495574 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":427,"threads_started":1}
I20260806 02:46:46.497473 11986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:46:46.580888 11991 heartbeater.cc:503] Master 127.1.210.126:34983 was elected leader, sending a full tablet report...
I20260806 02:46:46.582289 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 11864
W20260806 02:46:46.599833 11791 connection.cc:535] server connection from 127.1.210.65:49595 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 02:46:46.600039 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 11995
I20260806 02:46:46.604043 12265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:46.600979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39374 (local address 127.1.210.66:39883)
0806 02:46:46.601561 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:46.601568 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:46.601597 (+ 29us) server_negotiation.cc:407] Connection header received
0806 02:46:46.601671 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:46.601677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:46.601931 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:46.602068 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:46.603187 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:46.603195 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:46.603208 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:46:46.603236 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:46.603288 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:46:46.603305 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:46.603426 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:46.603596 (+ 170us) server_negotiation.cc:299] Negotiation successful
0806 02:46:46.603725 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":292,"threads_started":1}
I20260806 02:46:46.605353 12118 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:46:46.689771 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 11995
I20260806 02:46:46.708822 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 12126
I20260806 02:46:46.713251 12267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:46.710147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50076 (local address 127.1.210.67:38563)
0806 02:46:46.710963 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:46.710971 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:46.711005 (+ 34us) server_negotiation.cc:407] Connection header received
0806 02:46:46.711072 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:46.711080 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:46.711356 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:46.711497 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:46.712461 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:46.712468 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:46.712481 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:46:46.712507 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:46.712548 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:46:46.712562 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:46.712704 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:46.712769 (+ 65us) server_negotiation.cc:299] Negotiation successful
0806 02:46:46.712886 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":380,"threads_started":1}
I20260806 02:46:46.714649 12249 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:46:46.800124 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 12126
I20260806 02:46:46.818557 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 11774
I20260806 02:46:46.820000 11835 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:46:46.925832 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 11774
2026-08-06T02:46:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2577 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:46:47 dist-test-slave-8tdl krb5kdc[12272](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:46:47 dist-test-slave-8tdl krb5kdc[12272](info): set up 2 sockets
Aug 06 02:46:47 dist-test-slave-8tdl krb5kdc[12272](info): commencing operation
krb5kdc: starting...
W20260806 02:46:48.111287 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.005s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:46:48 dist-test-slave-8tdl krb5kdc[12272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:46:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:46:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
I20260806 02:46:48.400553 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:43995
--webserver_interface=127.1.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:38239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:43995
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:46:48.765826 12285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:48.766172 12285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:48.766268 12285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:48.774085 12285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:46:48.774222 12285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:48.774272 12285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:48.774323 12285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:46:48.774369 12285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:46:48.784952 12285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:38239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:43995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:43995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:46:48.786408 12285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:48.787796 12285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:48.801718 12293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:48.801846 12290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:48.801748 12291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:48.802736 12285 server_base.cc:1034] running on GCE node
I20260806 02:46:48.803656 12285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:48.805379 12285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:48.806726 12285 hybrid_clock.cc:648] HybridClock initialized: now 1785984408806613 us; error 49 us; skew 500 ppm
I20260806 02:46:48.807224 12285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:46:48 dist-test-slave-8tdl krb5kdc[12272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984408, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:48.811064 12285 init.cc:377] Logged in from keytab as kudu/127.1.210.126@KRBTEST.COM (short username kudu)
I20260806 02:46:48.812896 12285 webserver.cc:460] Webserver started at http://127.1.210.126:33749/ using document root <none> and password file <none>
I20260806 02:46:48.813717 12285 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:48.813905 12285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:48.814270 12285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:48.817937 12285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "548c70043c2348cb9373c187ffd716bd"
format_stamp: "Formatted at 2026-08-06 02:46:48 on dist-test-slave-8tdl"
server_key: "5ed937dcaf6c5ead73cda3e8ea393cc3"
server_key_iv: "81ea55394f85b00e1f46ea7aa5bfb16d"
server_key_version: "encryptionkey@0"
I20260806 02:46:48.819024 12285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "548c70043c2348cb9373c187ffd716bd"
format_stamp: "Formatted at 2026-08-06 02:46:48 on dist-test-slave-8tdl"
server_key: "5ed937dcaf6c5ead73cda3e8ea393cc3"
server_key_iv: "81ea55394f85b00e1f46ea7aa5bfb16d"
server_key_version: "encryptionkey@0"
I20260806 02:46:48.825796 12285 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260806 02:46:48.830024 12300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:48.831838 12285 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:46:48.832082 12285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "548c70043c2348cb9373c187ffd716bd"
format_stamp: "Formatted at 2026-08-06 02:46:48 on dist-test-slave-8tdl"
server_key: "5ed937dcaf6c5ead73cda3e8ea393cc3"
server_key_iv: "81ea55394f85b00e1f46ea7aa5bfb16d"
server_key_version: "encryptionkey@0"
I20260806 02:46:48.832365 12285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:48.851193 12285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:48.856104 12285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:48.856454 12285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:48.877629 12285 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:43995
I20260806 02:46:48.877668 12351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:43995 every 8 connection(s)
I20260806 02:46:48.879189 12285 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:46:48.881146 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 12285
I20260806 02:46:48.881402 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:46:48.882311 1865 external_mini_cluster.cc:1428] Setting key 74f31df68546748759e789c2c01316e9
I20260806 02:46:48.884712 12352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 02:46:48 dist-test-slave-8tdl krb5kdc[12272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:46:48.899880 12352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 548c70043c2348cb9373c187ffd716bd: Bootstrap starting.
I20260806 02:46:48.904649 12352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 548c70043c2348cb9373c187ffd716bd: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:48.906121 12352 log.cc:824] T 00000000000000000000000000000000 P 548c70043c2348cb9373c187ffd716bd: Log is configured to *not* fsync() on all Append() calls
I20260806 02:46:48.910379 12352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 548c70043c2348cb9373c187ffd716bd: No bootstrap required, opened a new log
I20260806 02:46:48.911535 12355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:48.884719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55950 (local address 127.1.210.126:43995)
0806 02:46:48.885663 (+ 944us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:48.885681 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:48.885737 (+ 56us) server_negotiation.cc:407] Connection header received
0806 02:46:48.888105 (+ 2368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:48.888139 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:48.888777 (+ 638us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:48.889301 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:48.891009 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:48.893154 (+ 2145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:48.894627 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:48.895848 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:48.896060 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:48.901106 (+ 5046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:48.901118 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:48.901139 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:48.901197 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:48.905623 (+ 4426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:48.906739 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:48.906755 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:48.906768 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:48.907166 (+ 398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:48.907663 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:48.907667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:48.907669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:48.908479 (+ 810us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:48.908839 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:48.909358 (+ 519us) server_negotiation.cc:299] Negotiation successful
0806 02:46:48.909908 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":395,"threads_started":1}
I20260806 02:46:48.919673 12352 raft_consensus.cc:348] T 00000000000000000000000000000000 P 548c70043c2348cb9373c187ffd716bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548c70043c2348cb9373c187ffd716bd" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 43995 } }
I20260806 02:46:48.920185 12352 raft_consensus.cc:374] T 00000000000000000000000000000000 P 548c70043c2348cb9373c187ffd716bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:48.920307 12352 raft_consensus.cc:729] T 00000000000000000000000000000000 P 548c70043c2348cb9373c187ffd716bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548c70043c2348cb9373c187ffd716bd, State: Initialized, Role: FOLLOWER
I20260806 02:46:48.920976 12352 consensus_queue.cc:260] T 00000000000000000000000000000000 P 548c70043c2348cb9373c187ffd716bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548c70043c2348cb9373c187ffd716bd" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 43995 } }
I20260806 02:46:48.921221 12352 raft_consensus.cc:388] T 00000000000000000000000000000000 P 548c70043c2348cb9373c187ffd716bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:46:48.921329 12352 raft_consensus.cc:482] T 00000000000000000000000000000000 P 548c70043c2348cb9373c187ffd716bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:46:48.921470 12352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 548c70043c2348cb9373c187ffd716bd [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:48.923849 12352 raft_consensus.cc:504] T 00000000000000000000000000000000 P 548c70043c2348cb9373c187ffd716bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548c70043c2348cb9373c187ffd716bd" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 43995 } }
I20260806 02:46:48.924510 12352 leader_election.cc:302] T 00000000000000000000000000000000 P 548c70043c2348cb9373c187ffd716bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 548c70043c2348cb9373c187ffd716bd; no voters:
I20260806 02:46:48.925351 12352 leader_election.cc:288] T 00000000000000000000000000000000 P 548c70043c2348cb9373c187ffd716bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:46:48.925503 12357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 548c70043c2348cb9373c187ffd716bd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:46:48.926360 12357 raft_consensus.cc:686] T 00000000000000000000000000000000 P 548c70043c2348cb9373c187ffd716bd [term 1 LEADER]: Becoming Leader. State: Replica: 548c70043c2348cb9373c187ffd716bd, State: Running, Role: LEADER
I20260806 02:46:48.927098 12357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 548c70043c2348cb9373c187ffd716bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548c70043c2348cb9373c187ffd716bd" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 43995 } }
I20260806 02:46:48.928167 12352 sys_catalog.cc:564] T 00000000000000000000000000000000 P 548c70043c2348cb9373c187ffd716bd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:46:48.930454 12359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 548c70043c2348cb9373c187ffd716bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "548c70043c2348cb9373c187ffd716bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548c70043c2348cb9373c187ffd716bd" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 43995 } } }
I20260806 02:46:48.930938 12359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 548c70043c2348cb9373c187ffd716bd [sys.catalog]: This master's current role is: LEADER
I20260806 02:46:48.932072 12358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 548c70043c2348cb9373c187ffd716bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 548c70043c2348cb9373c187ffd716bd. Latest consensus state: current_term: 1 leader_uuid: "548c70043c2348cb9373c187ffd716bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548c70043c2348cb9373c187ffd716bd" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 43995 } } }
I20260806 02:46:48.932528 12358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 548c70043c2348cb9373c187ffd716bd [sys.catalog]: This master's current role is: LEADER
I20260806 02:46:48.934621 12365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:46:48.940181 12365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:46:48.950616 12365 catalog_manager.cc:1347] Generated new cluster ID: a279108b166141e08209932e88df60c8
I20260806 02:46:48.950815 12365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:46:48.992684 12365 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:46:48.994261 12365 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:46:49.002555 12365 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 548c70043c2348cb9373c187ffd716bd: Generated new TSK 0
I20260806 02:46:49.003616 12365 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab.
Entry for principal kudu/127.1.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab.
I20260806 02:46:49.083503 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.65:0
--local_ip_for_outbound_sockets=127.1.210.65
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:43995
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:38239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:46:49.448338 12378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:49.448700 12378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:49.448884 12378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:49.457950 12378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:49.458169 12378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:49.458384 12378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.65
I20260806 02:46:49.469686 12378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:38239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:43995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.65
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:46:49.471220 12378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:49.472640 12378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:49.487399 12383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:49.487463 12384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:49.487412 12386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:49.488726 12378 server_base.cc:1034] running on GCE node
I20260806 02:46:49.489513 12378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:49.490594 12378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:49.491842 12378 hybrid_clock.cc:648] HybridClock initialized: now 1785984409491789 us; error 47 us; skew 500 ppm
I20260806 02:46:49.492231 12378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:46:49 dist-test-slave-8tdl krb5kdc[12272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984409, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:49.495925 12378 init.cc:377] Logged in from keytab as kudu/127.1.210.65@KRBTEST.COM (short username kudu)
I20260806 02:46:49.497674 12378 webserver.cc:460] Webserver started at http://127.1.210.65:34455/ using document root <none> and password file <none>
I20260806 02:46:49.498580 12378 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:49.498770 12378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:49.499109 12378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:49.502367 12378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/data/instance:
uuid: "dde73add689c439597720e3fafceb342"
format_stamp: "Formatted at 2026-08-06 02:46:49 on dist-test-slave-8tdl"
server_key: "b960bd75f5731390187732156c9a84e6"
server_key_iv: "4f73928c3417e855be161d21cc0a61ea"
server_key_version: "encryptionkey@0"
I20260806 02:46:49.503304 12378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance:
uuid: "dde73add689c439597720e3fafceb342"
format_stamp: "Formatted at 2026-08-06 02:46:49 on dist-test-slave-8tdl"
server_key: "b960bd75f5731390187732156c9a84e6"
server_key_iv: "4f73928c3417e855be161d21cc0a61ea"
server_key_version: "encryptionkey@0"
I20260806 02:46:49.509052 12378 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 02:46:49.512987 12393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:49.514446 12378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:46:49.514645 12378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/wal
uuid: "dde73add689c439597720e3fafceb342"
format_stamp: "Formatted at 2026-08-06 02:46:49 on dist-test-slave-8tdl"
server_key: "b960bd75f5731390187732156c9a84e6"
server_key_iv: "4f73928c3417e855be161d21cc0a61ea"
server_key_version: "encryptionkey@0"
I20260806 02:46:49.514950 12378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:49.529111 12378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:49.533882 12378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:49.534214 12378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:49.535442 12378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:46:49.536904 12378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:49.537029 12378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:49.537178 12378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:49.537257 12378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:49.574528 12378 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.65:43887
I20260806 02:46:49.574640 12505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.65:43887 every 8 connection(s)
I20260806 02:46:49.576025 12378 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
I20260806 02:46:49.584344 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 12378
I20260806 02:46:49.584597 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance
I20260806 02:46:49.585253 1865 external_mini_cluster.cc:1428] Setting key 934a975fdf5939ba325d183f46b0aecc
Aug 06 02:46:49 dist-test-slave-8tdl krb5kdc[12272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984409, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:46:49.599306 12508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:49.578881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:32785 (local address 127.1.210.126:43995)
0806 02:46:49.579375 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:49.579380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:49.580479 (+ 1099us) server_negotiation.cc:407] Connection header received
0806 02:46:49.582460 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:49.582466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:49.582574 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:49.582726 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:49.584373 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:49.585223 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:49.587153 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:49.587555 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:49.587694 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:49.592434 (+ 4740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:49.592441 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:49.592444 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:49.592481 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:49.595829 (+ 3348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:49.596711 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:49.596715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:49.596717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:49.596842 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:49.597394 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:49.597398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:49.597400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:49.597700 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:49.597861 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:49.598645 (+ 784us) server_negotiation.cc:299] Negotiation successful
0806 02:46:49.598913 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":246,"threads_started":1}
I20260806 02:46:49.600526 12509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:49.579494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:43995 (local address 127.1.210.65:32785)
0806 02:46:49.580283 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:49.580331 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:49.582156 (+ 1825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:49.582983 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:49.582999 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:49.583546 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:49.584179 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:49.584198 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:49.585409 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:49.585413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:49.586970 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:49.586986 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:49.587811 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:49.587818 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:49.588410 (+ 592us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:49.589198 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:49.589222 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:49.592120 (+ 2898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:49.596024 (+ 3904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:49.596045 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:49.596067 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:49.596554 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:49.596986 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:49.596991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:49.596995 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:49.597256 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:49.597839 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:49.597848 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:49.598239 (+ 391us) client_negotiation.cc:769] Sending connection context
0806 02:46:49.598519 (+ 280us) client_negotiation.cc:240] Negotiation successful
0806 02:46:49.599161 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":315,"threads_started":1}
I20260806 02:46:49.602263 12506 heartbeater.cc:347] Connected to a master server at 127.1.210.126:43995
I20260806 02:46:49.602632 12506 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:49.603670 12506 heartbeater.cc:511] Master 127.1.210.126:43995 requested a full tablet report, sending...
I20260806 02:46:49.606609 12317 ts_manager.cc:194] Registered new tserver with Master: dde73add689c439597720e3fafceb342 (127.1.210.65:43887)
I20260806 02:46:49.608305 12317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.65@KRBTEST.COM'} at 127.1.210.65:32785
WARNING: no policy specified for kudu/127.1.210.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab.
Entry for principal kudu/127.1.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab.
I20260806 02:46:49.651428 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.66:0
--local_ip_for_outbound_sockets=127.1.210.66
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:43995
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:38239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:46:50.012352 12512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:50.012677 12512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:50.012837 12512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:50.021015 12512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:50.021152 12512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:50.021329 12512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.66
I20260806 02:46:50.032701 12512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:38239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:43995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.66
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:46:50.034726 12512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:50.036247 12512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:50.051620 12517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:50.051674 12518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:50.051810 12520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:50.053754 12512 server_base.cc:1034] running on GCE node
I20260806 02:46:50.054772 12512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:50.056005 12512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:50.057294 12512 hybrid_clock.cc:648] HybridClock initialized: now 1785984410057247 us; error 44 us; skew 500 ppm
I20260806 02:46:50.057762 12512 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:46:50 dist-test-slave-8tdl krb5kdc[12272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984410, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:50.062099 12512 init.cc:377] Logged in from keytab as kudu/127.1.210.66@KRBTEST.COM (short username kudu)
I20260806 02:46:50.063978 12512 webserver.cc:460] Webserver started at http://127.1.210.66:41309/ using document root <none> and password file <none>
I20260806 02:46:50.064772 12512 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:50.064935 12512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:50.065271 12512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:50.068477 12512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/data/instance:
uuid: "51386e0b5b304a50b1e7fc6a8a7d6447"
format_stamp: "Formatted at 2026-08-06 02:46:50 on dist-test-slave-8tdl"
server_key: "fc9a6b54e29476bde23dbc0d8fc3bf91"
server_key_iv: "a64b0b83ea3e4e14631d532fa7bbbcd6"
server_key_version: "encryptionkey@0"
I20260806 02:46:50.069344 12512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance:
uuid: "51386e0b5b304a50b1e7fc6a8a7d6447"
format_stamp: "Formatted at 2026-08-06 02:46:50 on dist-test-slave-8tdl"
server_key: "fc9a6b54e29476bde23dbc0d8fc3bf91"
server_key_iv: "a64b0b83ea3e4e14631d532fa7bbbcd6"
server_key_version: "encryptionkey@0"
I20260806 02:46:50.075516 12512 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260806 02:46:50.079783 12527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:50.081321 12512 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260806 02:46:50.081576 12512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/wal
uuid: "51386e0b5b304a50b1e7fc6a8a7d6447"
format_stamp: "Formatted at 2026-08-06 02:46:50 on dist-test-slave-8tdl"
server_key: "fc9a6b54e29476bde23dbc0d8fc3bf91"
server_key_iv: "a64b0b83ea3e4e14631d532fa7bbbcd6"
server_key_version: "encryptionkey@0"
I20260806 02:46:50.081854 12512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:50.105087 12512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:50.110309 12512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:50.110605 12512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:50.111824 12512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:46:50.113317 12512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:50.113401 12512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:50.113497 12512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:50.113545 12512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:50.152413 12512 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.66:39313
I20260806 02:46:50.152477 12640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.66:39313 every 8 connection(s)
I20260806 02:46:50.153865 12512 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
I20260806 02:46:50.161257 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 12512
I20260806 02:46:50.161484 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance
I20260806 02:46:50.161969 1865 external_mini_cluster.cc:1428] Setting key d6b0417ec8be5c97c8179627a5e995bb
Aug 06 02:46:50 dist-test-slave-8tdl krb5kdc[12272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984410, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:46:50.180804 12643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:50.156823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:57067 (local address 127.1.210.126:43995)
0806 02:46:50.157333 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:50.157338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:50.158422 (+ 1084us) server_negotiation.cc:407] Connection header received
0806 02:46:50.160557 (+ 2135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:50.160563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:50.160659 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:50.161065 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:50.163364 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:50.164430 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:50.166396 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:50.167260 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:50.167430 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:50.172907 (+ 5477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:50.172917 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:50.172924 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:50.173059 (+ 135us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:50.176599 (+ 3540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:50.177745 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:50.177750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:50.177753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:50.177928 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:50.178545 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:50.178550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:50.178552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:50.178996 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:50.179171 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:50.180303 (+ 1132us) server_negotiation.cc:299] Negotiation successful
0806 02:46:50.180522 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":238,"threads_started":1}
I20260806 02:46:50.182461 12644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:50.157479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:43995 (local address 127.1.210.66:57067)
0806 02:46:50.158238 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:50.158289 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:50.160235 (+ 1946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:50.161372 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:50.161389 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:50.162144 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:50.163004 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:50.163039 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:50.164704 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:50.164711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:50.166200 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:50.166216 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:50.167555 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:50.167561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:50.167808 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:50.168654 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:50.168682 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:50.172532 (+ 3850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:50.176800 (+ 4268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:50.176811 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:50.176840 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:50.177545 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:50.178065 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:50.178073 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:50.178079 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:50.178374 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:50.179151 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:50.179162 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:50.179714 (+ 552us) client_negotiation.cc:769] Sending connection context
0806 02:46:50.180132 (+ 418us) client_negotiation.cc:240] Negotiation successful
0806 02:46:50.180802 (+ 670us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":296,"threads_started":1}
I20260806 02:46:50.184229 12641 heartbeater.cc:347] Connected to a master server at 127.1.210.126:43995
I20260806 02:46:50.184584 12641 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:50.185600 12641 heartbeater.cc:511] Master 127.1.210.126:43995 requested a full tablet report, sending...
I20260806 02:46:50.187713 12317 ts_manager.cc:194] Registered new tserver with Master: 51386e0b5b304a50b1e7fc6a8a7d6447 (127.1.210.66:39313)
I20260806 02:46:50.189214 12317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.66@KRBTEST.COM'} at 127.1.210.66:57067
WARNING: no policy specified for kudu/127.1.210.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab.
Entry for principal kudu/127.1.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab.
I20260806 02:46:50.235363 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.67:0
--local_ip_for_outbound_sockets=127.1.210.67
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:43995
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:38239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:46:50.608371 12647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:50.608700 12647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:50.608860 12647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 02:46:50.612828 12506 heartbeater.cc:503] Master 127.1.210.126:43995 was elected leader, sending a full tablet report...
W20260806 02:46:50.617226 12647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:50.617353 12647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:50.617589 12647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.67
I20260806 02:46:50.628046 12647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:38239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:43995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.67
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:46:50.629596 12647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:50.631073 12647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:50.645149 12652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:50.645431 12653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:50.645455 12655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:50.646800 12647 server_base.cc:1034] running on GCE node
I20260806 02:46:50.647447 12647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:50.648468 12647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:50.649755 12647 hybrid_clock.cc:648] HybridClock initialized: now 1785984410649710 us; error 38 us; skew 500 ppm
I20260806 02:46:50.650133 12647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:46:50 dist-test-slave-8tdl krb5kdc[12272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984410, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:50.653895 12647 init.cc:377] Logged in from keytab as kudu/127.1.210.67@KRBTEST.COM (short username kudu)
I20260806 02:46:50.655675 12647 webserver.cc:460] Webserver started at http://127.1.210.67:43029/ using document root <none> and password file <none>
I20260806 02:46:50.656435 12647 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:50.656600 12647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:50.656926 12647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:50.660171 12647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/data/instance:
uuid: "4e7ad3026ef0490d8f87163cf83288c1"
format_stamp: "Formatted at 2026-08-06 02:46:50 on dist-test-slave-8tdl"
server_key: "37bd852f81716b3567835ada328fcc69"
server_key_iv: "e9b2fe7a2ecff3a73e1d824f87ee254c"
server_key_version: "encryptionkey@0"
I20260806 02:46:50.661000 12647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance:
uuid: "4e7ad3026ef0490d8f87163cf83288c1"
format_stamp: "Formatted at 2026-08-06 02:46:50 on dist-test-slave-8tdl"
server_key: "37bd852f81716b3567835ada328fcc69"
server_key_iv: "e9b2fe7a2ecff3a73e1d824f87ee254c"
server_key_version: "encryptionkey@0"
I20260806 02:46:50.666625 12647 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 02:46:50.670780 12662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:50.672442 12647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:46:50.672614 12647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/wal
uuid: "4e7ad3026ef0490d8f87163cf83288c1"
format_stamp: "Formatted at 2026-08-06 02:46:50 on dist-test-slave-8tdl"
server_key: "37bd852f81716b3567835ada328fcc69"
server_key_iv: "e9b2fe7a2ecff3a73e1d824f87ee254c"
server_key_version: "encryptionkey@0"
I20260806 02:46:50.672817 12647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:50.689268 12647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:50.693833 12647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:50.694175 12647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:50.695330 12647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:46:50.696822 12647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:50.696931 12647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:50.697099 12647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:50.697173 12647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:50.731192 12647 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.67:46449
I20260806 02:46:50.731279 12774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.67:46449 every 8 connection(s)
I20260806 02:46:50.732580 12647 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
I20260806 02:46:50.735090 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 12647
I20260806 02:46:50.735332 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance
I20260806 02:46:50.735877 1865 external_mini_cluster.cc:1428] Setting key 1d97af05ab5b411f4da970f018a5e643
Aug 06 02:46:50 dist-test-slave-8tdl krb5kdc[12272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984410, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:46:50.756507 12777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:50.735627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:42163 (local address 127.1.210.126:43995)
0806 02:46:50.736183 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:50.736192 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:50.737879 (+ 1687us) server_negotiation.cc:407] Connection header received
0806 02:46:50.740231 (+ 2352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:50.740236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:50.740303 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:50.740399 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:50.742123 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:50.743030 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:50.744665 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:50.745077 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:50.745178 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:50.749062 (+ 3884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:50.749068 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:50.749071 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:50.749109 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:50.752904 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:50.753854 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:50.753859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:50.753861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:50.754061 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:50.754573 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:50.754577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:50.754579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:50.755024 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:50.755160 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:50.756016 (+ 856us) server_negotiation.cc:299] Negotiation successful
0806 02:46:50.756204 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":333,"threads_started":1}
I20260806 02:46:50.757568 12778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:50.736542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:43995 (local address 127.1.210.67:42163)
0806 02:46:50.737688 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:50.737745 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:50.739953 (+ 2208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:50.740672 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:50.740681 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:50.741281 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:50.741875 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:50.741903 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:50.743210 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:50.743215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:50.744517 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:50.744530 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:50.745288 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:50.745293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:50.745513 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:50.746188 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:50.746211 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:50.748771 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:50.753161 (+ 4390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:50.753170 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:50.753187 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:50.753719 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:50.754195 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:50.754200 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:50.754202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:50.754430 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:50.755167 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:50.755174 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:50.755553 (+ 379us) client_negotiation.cc:769] Sending connection context
0806 02:46:50.755835 (+ 282us) client_negotiation.cc:240] Negotiation successful
0806 02:46:50.756287 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":447,"threads_started":1}
I20260806 02:46:50.759181 12775 heartbeater.cc:347] Connected to a master server at 127.1.210.126:43995
I20260806 02:46:50.759574 12775 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:50.760402 12775 heartbeater.cc:511] Master 127.1.210.126:43995 requested a full tablet report, sending...
I20260806 02:46:50.762125 12317 ts_manager.cc:194] Registered new tserver with Master: 4e7ad3026ef0490d8f87163cf83288c1 (127.1.210.67:46449)
I20260806 02:46:50.763134 12317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.67@KRBTEST.COM'} at 127.1.210.67:42163
I20260806 02:46:50.765292 1865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:46:50.784019 12777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:50.770166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55960 (local address 127.1.210.126:43995)
0806 02:46:50.770369 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:50.770378 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:50.770880 (+ 502us) server_negotiation.cc:407] Connection header received
0806 02:46:50.771152 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:50.771161 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:50.771229 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:50.771337 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:50.772781 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:50.773520 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:50.774907 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:50.775324 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:50.775472 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:50.778184 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:50.778189 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:50.778192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:50.778230 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:50.781112 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:50.781821 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:50.781828 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:50.781832 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:50.781965 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:50.782378 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:50.782384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:50.782387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:50.782842 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:50.783136 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:50.783443 (+ 307us) server_negotiation.cc:299] Negotiation successful
0806 02:46:50.783681 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:46:50.788637 12317 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:46:50.795807 12317 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:46:50.819805 12789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:50.810068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59762 (local address 127.1.210.66:39313)
0806 02:46:50.811261 (+ 1193us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:50.811270 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:50.811298 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:46:50.811416 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:50.811424 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:50.811715 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:50.811930 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:46:50.812760 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:50.814193 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:50.817239 (+ 3046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:50.818336 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:50.818542 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:50.818809 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:50.819255 (+ 446us) server_negotiation.cc:299] Negotiation successful
0806 02:46:50.819434 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1023,"thread_start_us":1208,"threads_started":1}
I20260806 02:46:50.819886 12788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:50.809781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:39313 (local address 127.0.0.1:59762)
0806 02:46:50.810646 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:50.810725 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:50.810940 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:50.811966 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:50.811974 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:50.812017 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:46:50.812497 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:50.812512 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:50.814368 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:50.814379 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:50.817068 (+ 2689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:50.817085 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:50.818612 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:50.818618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:50.818906 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:50.818971 (+ 65us) client_negotiation.cc:769] Sending connection context
0806 02:46:50.819202 (+ 231us) client_negotiation.cc:240] Negotiation successful
0806 02:46:50.819429 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":25,"thread_start_us":349,"threads_started":1}
I20260806 02:46:50.822618 12793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:50.811819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:46449 (local address 127.0.0.1:60580)
0806 02:46:50.813304 (+ 1485us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:50.813319 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:50.813460 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:50.814196 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:50.814201 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:50.814229 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:46:50.814822 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:50.814840 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:50.816819 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:50.816824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:50.819428 (+ 2604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:50.819455 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:50.821705 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:50.821711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:50.821887 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:50.821922 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 02:46:50.822014 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 02:46:50.822160 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1375,"mutex_wait_us":7,"spinlock_wait_cycles":7680,"thread_start_us":1350,"threads_started":1}
I20260806 02:46:50.822634 12791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:50.810962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60580 (local address 127.1.210.67:46449)
0806 02:46:50.811737 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:50.811745 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:50.813589 (+ 1844us) server_negotiation.cc:407] Connection header received
0806 02:46:50.813800 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:50.813806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:50.813980 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:50.814190 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:46:50.815112 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:50.816649 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:50.819652 (+ 3003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:50.821181 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:50.821319 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:50.821563 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:50.822127 (+ 564us) server_negotiation.cc:299] Negotiation successful
0806 02:46:50.822310 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":504,"threads_started":1}
I20260806 02:46:50.823815 12790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:50.810668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:43887 (local address 127.0.0.1:39256)
0806 02:46:50.813276 (+ 2608us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:50.813304 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:50.813455 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:50.814499 (+ 1044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:50.814505 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:50.814526 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:46:50.815140 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:50.815154 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:50.817145 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:50.817151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:50.820528 (+ 3377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:50.820541 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:50.822944 (+ 2403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:50.822954 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:50.823172 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:50.823209 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 02:46:50.823307 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 02:46:50.823449 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2433,"spinlock_wait_cycles":10112,"thread_start_us":2191,"threads_started":1}
I20260806 02:46:50.824286 12792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:50.811795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39256 (local address 127.1.210.65:43887)
0806 02:46:50.812726 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:50.812737 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:50.813589 (+ 852us) server_negotiation.cc:407] Connection header received
0806 02:46:50.813697 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:50.813704 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:50.813986 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:50.814190 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:46:50.815373 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:50.816982 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:50.820727 (+ 3745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:50.822302 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:50.822484 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:50.822782 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:50.823399 (+ 617us) server_negotiation.cc:299] Negotiation successful
0806 02:46:50.823581 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":770,"thread_start_us":602,"threads_started":1}
I20260806 02:46:50.824482 12576 tablet_service.cc:1467] Processing CreateTablet for tablet fa4f13de5aaf4fd8899c1c1931251308 (DEFAULT_TABLE table=test-table [id=f7cbc640ce6f434d804ed1770a779dca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:46:50.825832 12576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa4f13de5aaf4fd8899c1c1931251308. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:50.826617 12710 tablet_service.cc:1467] Processing CreateTablet for tablet fa4f13de5aaf4fd8899c1c1931251308 (DEFAULT_TABLE table=test-table [id=f7cbc640ce6f434d804ed1770a779dca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:46:50.828276 12710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa4f13de5aaf4fd8899c1c1931251308. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:50.830665 12441 tablet_service.cc:1467] Processing CreateTablet for tablet fa4f13de5aaf4fd8899c1c1931251308 (DEFAULT_TABLE table=test-table [id=f7cbc640ce6f434d804ed1770a779dca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:46:50.832350 12441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa4f13de5aaf4fd8899c1c1931251308. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:50.842203 12794 tablet_bootstrap.cc:492] T fa4f13de5aaf4fd8899c1c1931251308 P 4e7ad3026ef0490d8f87163cf83288c1: Bootstrap starting.
I20260806 02:46:50.847007 12794 tablet_bootstrap.cc:654] T fa4f13de5aaf4fd8899c1c1931251308 P 4e7ad3026ef0490d8f87163cf83288c1: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:50.848155 12794 log.cc:824] T fa4f13de5aaf4fd8899c1c1931251308 P 4e7ad3026ef0490d8f87163cf83288c1: Log is configured to *not* fsync() on all Append() calls
I20260806 02:46:50.848227 12795 tablet_bootstrap.cc:492] T fa4f13de5aaf4fd8899c1c1931251308 P 51386e0b5b304a50b1e7fc6a8a7d6447: Bootstrap starting.
I20260806 02:46:50.852283 12794 tablet_bootstrap.cc:492] T fa4f13de5aaf4fd8899c1c1931251308 P 4e7ad3026ef0490d8f87163cf83288c1: No bootstrap required, opened a new log
I20260806 02:46:50.852609 12794 ts_tablet_manager.cc:1397] T fa4f13de5aaf4fd8899c1c1931251308 P 4e7ad3026ef0490d8f87163cf83288c1: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.006s
I20260806 02:46:50.853149 12795 tablet_bootstrap.cc:654] T fa4f13de5aaf4fd8899c1c1931251308 P 51386e0b5b304a50b1e7fc6a8a7d6447: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:50.854969 12795 log.cc:824] T fa4f13de5aaf4fd8899c1c1931251308 P 51386e0b5b304a50b1e7fc6a8a7d6447: Log is configured to *not* fsync() on all Append() calls
I20260806 02:46:50.855880 12797 tablet_bootstrap.cc:492] T fa4f13de5aaf4fd8899c1c1931251308 P dde73add689c439597720e3fafceb342: Bootstrap starting.
I20260806 02:46:50.858626 12795 tablet_bootstrap.cc:492] T fa4f13de5aaf4fd8899c1c1931251308 P 51386e0b5b304a50b1e7fc6a8a7d6447: No bootstrap required, opened a new log
I20260806 02:46:50.859056 12795 ts_tablet_manager.cc:1397] T fa4f13de5aaf4fd8899c1c1931251308 P 51386e0b5b304a50b1e7fc6a8a7d6447: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260806 02:46:50.859823 12797 tablet_bootstrap.cc:654] T fa4f13de5aaf4fd8899c1c1931251308 P dde73add689c439597720e3fafceb342: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:50.861295 12797 log.cc:824] T fa4f13de5aaf4fd8899c1c1931251308 P dde73add689c439597720e3fafceb342: Log is configured to *not* fsync() on all Append() calls
I20260806 02:46:50.862862 12794 raft_consensus.cc:348] T fa4f13de5aaf4fd8899c1c1931251308 P 4e7ad3026ef0490d8f87163cf83288c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51386e0b5b304a50b1e7fc6a8a7d6447" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 39313 } } peers { permanent_uuid: "4e7ad3026ef0490d8f87163cf83288c1" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 46449 } } peers { permanent_uuid: "dde73add689c439597720e3fafceb342" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 43887 } }
I20260806 02:46:50.863456 12794 raft_consensus.cc:374] T fa4f13de5aaf4fd8899c1c1931251308 P 4e7ad3026ef0490d8f87163cf83288c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:50.863584 12794 raft_consensus.cc:729] T fa4f13de5aaf4fd8899c1c1931251308 P 4e7ad3026ef0490d8f87163cf83288c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e7ad3026ef0490d8f87163cf83288c1, State: Initialized, Role: FOLLOWER
I20260806 02:46:50.864385 12794 consensus_queue.cc:260] T fa4f13de5aaf4fd8899c1c1931251308 P 4e7ad3026ef0490d8f87163cf83288c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51386e0b5b304a50b1e7fc6a8a7d6447" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 39313 } } peers { permanent_uuid: "4e7ad3026ef0490d8f87163cf83288c1" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 46449 } } peers { permanent_uuid: "dde73add689c439597720e3fafceb342" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 43887 } }
I20260806 02:46:50.865434 12797 tablet_bootstrap.cc:492] T fa4f13de5aaf4fd8899c1c1931251308 P dde73add689c439597720e3fafceb342: No bootstrap required, opened a new log
I20260806 02:46:50.865767 12797 ts_tablet_manager.cc:1397] T fa4f13de5aaf4fd8899c1c1931251308 P dde73add689c439597720e3fafceb342: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260806 02:46:50.865861 12775 heartbeater.cc:503] Master 127.1.210.126:43995 was elected leader, sending a full tablet report...
I20260806 02:46:50.866534 12794 ts_tablet_manager.cc:1428] T fa4f13de5aaf4fd8899c1c1931251308 P 4e7ad3026ef0490d8f87163cf83288c1: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260806 02:46:50.869582 12795 raft_consensus.cc:348] T fa4f13de5aaf4fd8899c1c1931251308 P 51386e0b5b304a50b1e7fc6a8a7d6447 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51386e0b5b304a50b1e7fc6a8a7d6447" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 39313 } } peers { permanent_uuid: "4e7ad3026ef0490d8f87163cf83288c1" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 46449 } } peers { permanent_uuid: "dde73add689c439597720e3fafceb342" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 43887 } }
I20260806 02:46:50.870023 12795 raft_consensus.cc:374] T fa4f13de5aaf4fd8899c1c1931251308 P 51386e0b5b304a50b1e7fc6a8a7d6447 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:50.870152 12795 raft_consensus.cc:729] T fa4f13de5aaf4fd8899c1c1931251308 P 51386e0b5b304a50b1e7fc6a8a7d6447 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51386e0b5b304a50b1e7fc6a8a7d6447, State: Initialized, Role: FOLLOWER
I20260806 02:46:50.870932 12795 consensus_queue.cc:260] T fa4f13de5aaf4fd8899c1c1931251308 P 51386e0b5b304a50b1e7fc6a8a7d6447 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51386e0b5b304a50b1e7fc6a8a7d6447" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 39313 } } peers { permanent_uuid: "4e7ad3026ef0490d8f87163cf83288c1" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 46449 } } peers { permanent_uuid: "dde73add689c439597720e3fafceb342" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 43887 } }
I20260806 02:46:50.875860 12797 raft_consensus.cc:348] T fa4f13de5aaf4fd8899c1c1931251308 P dde73add689c439597720e3fafceb342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51386e0b5b304a50b1e7fc6a8a7d6447" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 39313 } } peers { permanent_uuid: "4e7ad3026ef0490d8f87163cf83288c1" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 46449 } } peers { permanent_uuid: "dde73add689c439597720e3fafceb342" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 43887 } }
I20260806 02:46:50.876297 12797 raft_consensus.cc:374] T fa4f13de5aaf4fd8899c1c1931251308 P dde73add689c439597720e3fafceb342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:50.876493 12797 raft_consensus.cc:729] T fa4f13de5aaf4fd8899c1c1931251308 P dde73add689c439597720e3fafceb342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dde73add689c439597720e3fafceb342, State: Initialized, Role: FOLLOWER
I20260806 02:46:50.877223 12641 heartbeater.cc:503] Master 127.1.210.126:43995 was elected leader, sending a full tablet report...
I20260806 02:46:50.877226 12797 consensus_queue.cc:260] T fa4f13de5aaf4fd8899c1c1931251308 P dde73add689c439597720e3fafceb342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51386e0b5b304a50b1e7fc6a8a7d6447" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 39313 } } peers { permanent_uuid: "4e7ad3026ef0490d8f87163cf83288c1" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 46449 } } peers { permanent_uuid: "dde73add689c439597720e3fafceb342" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 43887 } }
I20260806 02:46:50.880127 12797 ts_tablet_manager.cc:1428] T fa4f13de5aaf4fd8899c1c1931251308 P dde73add689c439597720e3fafceb342: Time spent starting tablet: real 0.014s user 0.007s sys 0.006s
I20260806 02:46:50.881263 12795 ts_tablet_manager.cc:1428] T fa4f13de5aaf4fd8899c1c1931251308 P 51386e0b5b304a50b1e7fc6a8a7d6447: Time spent starting tablet: real 0.022s user 0.009s sys 0.008s
W20260806 02:46:50.906368 12642 tablet.cc:2369] T fa4f13de5aaf4fd8899c1c1931251308 P 51386e0b5b304a50b1e7fc6a8a7d6447: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:46:50.984377 12776 tablet.cc:2369] T fa4f13de5aaf4fd8899c1c1931251308 P 4e7ad3026ef0490d8f87163cf83288c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:46:51.073432 12800 raft_consensus.cc:482] T fa4f13de5aaf4fd8899c1c1931251308 P 4e7ad3026ef0490d8f87163cf83288c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:46:51.073756 12800 raft_consensus.cc:504] T fa4f13de5aaf4fd8899c1c1931251308 P 4e7ad3026ef0490d8f87163cf83288c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51386e0b5b304a50b1e7fc6a8a7d6447" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 39313 } } peers { permanent_uuid: "4e7ad3026ef0490d8f87163cf83288c1" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 46449 } } peers { permanent_uuid: "dde73add689c439597720e3fafceb342" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 43887 } }
I20260806 02:46:51.075700 12800 leader_election.cc:288] T fa4f13de5aaf4fd8899c1c1931251308 P 4e7ad3026ef0490d8f87163cf83288c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51386e0b5b304a50b1e7fc6a8a7d6447 (127.1.210.66:39313), dde73add689c439597720e3fafceb342 (127.1.210.65:43887)
W20260806 02:46:51.078857 12507 tablet.cc:2369] T fa4f13de5aaf4fd8899c1c1931251308 P dde73add689c439597720e3fafceb342: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:46:51.082844 12789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:51.076195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:57811 (local address 127.1.210.66:39313)
0806 02:46:51.076459 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:51.076466 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:51.076489 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:46:51.076575 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:51.076580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:51.076659 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:51.076760 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:46:51.077366 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:51.078412 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:51.080481 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:51.081829 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:51.081954 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:51.082013 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:51.082460 (+ 447us) server_negotiation.cc:299] Negotiation successful
0806 02:46:51.082564 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 02:46:51.082862 12778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:51.075903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:39313 (local address 127.1.210.67:57811)
0806 02:46:51.076156 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:51.076178 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:51.076312 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:51.076947 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:51.076951 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:51.076980 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:46:51.077192 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:51.077205 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:51.078561 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:51.078567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:51.080282 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:51.080295 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:51.082009 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:51.082015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:51.082229 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:51.082268 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 02:46:51.082381 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 02:46:51.082526 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 02:46:51.083895 12596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa4f13de5aaf4fd8899c1c1931251308" candidate_uuid: "4e7ad3026ef0490d8f87163cf83288c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51386e0b5b304a50b1e7fc6a8a7d6447" is_pre_election: true
I20260806 02:46:51.084255 12803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:51.076236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:43887 (local address 127.1.210.67:41721)
0806 02:46:51.077087 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:51.077101 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:51.077233 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:51.077543 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:51.077547 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:51.077572 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:46:51.077958 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:51.077970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:51.079675 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:51.079680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:51.081582 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:51.081600 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:51.083440 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:51.083446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:51.083633 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:51.083655 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:46:51.083768 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 02:46:51.083897 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":494,"threads_started":1}
I20260806 02:46:51.084573 12792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:51.076633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:41721 (local address 127.1.210.65:43887)
0806 02:46:51.076834 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:51.076840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:51.077260 (+ 420us) server_negotiation.cc:407] Connection header received
0806 02:46:51.077343 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:51.077348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:51.077421 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:51.077489 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:46:51.078222 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:51.079477 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:51.081797 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:51.083220 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:51.083420 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:51.083492 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:51.083799 (+ 307us) server_negotiation.cc:299] Negotiation successful
0806 02:46:51.084294 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 02:46:51.084510 12596 raft_consensus.cc:2454] T fa4f13de5aaf4fd8899c1c1931251308 P 51386e0b5b304a50b1e7fc6a8a7d6447 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e7ad3026ef0490d8f87163cf83288c1 in term 0.
I20260806 02:46:51.085325 12666 leader_election.cc:302] T fa4f13de5aaf4fd8899c1c1931251308 P 4e7ad3026ef0490d8f87163cf83288c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e7ad3026ef0490d8f87163cf83288c1, 51386e0b5b304a50b1e7fc6a8a7d6447; no voters:
I20260806 02:46:51.085736 12800 raft_consensus.cc:2781] T fa4f13de5aaf4fd8899c1c1931251308 P 4e7ad3026ef0490d8f87163cf83288c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:46:51.085551 12461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa4f13de5aaf4fd8899c1c1931251308" candidate_uuid: "4e7ad3026ef0490d8f87163cf83288c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dde73add689c439597720e3fafceb342" is_pre_election: true
I20260806 02:46:51.085908 12800 raft_consensus.cc:482] T fa4f13de5aaf4fd8899c1c1931251308 P 4e7ad3026ef0490d8f87163cf83288c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:46:51.086037 12800 raft_consensus.cc:3037] T fa4f13de5aaf4fd8899c1c1931251308 P 4e7ad3026ef0490d8f87163cf83288c1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:51.086205 12461 raft_consensus.cc:2454] T fa4f13de5aaf4fd8899c1c1931251308 P dde73add689c439597720e3fafceb342 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e7ad3026ef0490d8f87163cf83288c1 in term 0.
I20260806 02:46:51.088711 12800 raft_consensus.cc:504] T fa4f13de5aaf4fd8899c1c1931251308 P 4e7ad3026ef0490d8f87163cf83288c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51386e0b5b304a50b1e7fc6a8a7d6447" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 39313 } } peers { permanent_uuid: "4e7ad3026ef0490d8f87163cf83288c1" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 46449 } } peers { permanent_uuid: "dde73add689c439597720e3fafceb342" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 43887 } }
I20260806 02:46:51.089354 12800 leader_election.cc:288] T fa4f13de5aaf4fd8899c1c1931251308 P 4e7ad3026ef0490d8f87163cf83288c1 [CANDIDATE]: Term 1 election: Requested vote from peers 51386e0b5b304a50b1e7fc6a8a7d6447 (127.1.210.66:39313), dde73add689c439597720e3fafceb342 (127.1.210.65:43887)
I20260806 02:46:51.089879 12461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa4f13de5aaf4fd8899c1c1931251308" candidate_uuid: "4e7ad3026ef0490d8f87163cf83288c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dde73add689c439597720e3fafceb342"
I20260806 02:46:51.089879 12596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa4f13de5aaf4fd8899c1c1931251308" candidate_uuid: "4e7ad3026ef0490d8f87163cf83288c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51386e0b5b304a50b1e7fc6a8a7d6447"
I20260806 02:46:51.090109 12461 raft_consensus.cc:3037] T fa4f13de5aaf4fd8899c1c1931251308 P dde73add689c439597720e3fafceb342 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:51.090119 12596 raft_consensus.cc:3037] T fa4f13de5aaf4fd8899c1c1931251308 P 51386e0b5b304a50b1e7fc6a8a7d6447 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:51.092931 12596 raft_consensus.cc:2454] T fa4f13de5aaf4fd8899c1c1931251308 P 51386e0b5b304a50b1e7fc6a8a7d6447 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e7ad3026ef0490d8f87163cf83288c1 in term 1.
I20260806 02:46:51.093254 12461 raft_consensus.cc:2454] T fa4f13de5aaf4fd8899c1c1931251308 P dde73add689c439597720e3fafceb342 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e7ad3026ef0490d8f87163cf83288c1 in term 1.
I20260806 02:46:51.093571 12666 leader_election.cc:302] T fa4f13de5aaf4fd8899c1c1931251308 P 4e7ad3026ef0490d8f87163cf83288c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e7ad3026ef0490d8f87163cf83288c1, 51386e0b5b304a50b1e7fc6a8a7d6447; no voters:
I20260806 02:46:51.093943 12800 raft_consensus.cc:2781] T fa4f13de5aaf4fd8899c1c1931251308 P 4e7ad3026ef0490d8f87163cf83288c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:46:51.094436 12800 raft_consensus.cc:686] T fa4f13de5aaf4fd8899c1c1931251308 P 4e7ad3026ef0490d8f87163cf83288c1 [term 1 LEADER]: Becoming Leader. State: Replica: 4e7ad3026ef0490d8f87163cf83288c1, State: Running, Role: LEADER
I20260806 02:46:51.095134 12800 consensus_queue.cc:237] T fa4f13de5aaf4fd8899c1c1931251308 P 4e7ad3026ef0490d8f87163cf83288c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51386e0b5b304a50b1e7fc6a8a7d6447" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 39313 } } peers { permanent_uuid: "4e7ad3026ef0490d8f87163cf83288c1" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 46449 } } peers { permanent_uuid: "dde73add689c439597720e3fafceb342" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 43887 } }
I20260806 02:46:51.100369 12316 catalog_manager.cc:5495] T fa4f13de5aaf4fd8899c1c1931251308 P 4e7ad3026ef0490d8f87163cf83288c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e7ad3026ef0490d8f87163cf83288c1 (127.1.210.67). New cstate: current_term: 1 leader_uuid: "4e7ad3026ef0490d8f87163cf83288c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51386e0b5b304a50b1e7fc6a8a7d6447" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 39313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e7ad3026ef0490d8f87163cf83288c1" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 46449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dde73add689c439597720e3fafceb342" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 43887 } health_report { overall_health: UNKNOWN } } }
I20260806 02:46:51.133759 12791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:51.126829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60594 (local address 127.1.210.67:46449)
0806 02:46:51.127155 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:51.127164 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:51.127221 (+ 57us) server_negotiation.cc:407] Connection header received
0806 02:46:51.127444 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:51.127448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:51.127555 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:51.127698 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:46:51.128312 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:51.129250 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:51.131049 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:51.131494 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:51.131640 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:51.132262 (+ 622us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:46:51.132966 (+ 704us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:46:51.133123 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:51.133325 (+ 202us) server_negotiation.cc:299] Negotiation successful
0806 02:46:51.133418 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260806 02:46:51.143167 12461 raft_consensus.cc:1261] T fa4f13de5aaf4fd8899c1c1931251308 P dde73add689c439597720e3fafceb342 [term 1 FOLLOWER]: Refusing update from remote peer 4e7ad3026ef0490d8f87163cf83288c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:46:51.143167 12596 raft_consensus.cc:1261] T fa4f13de5aaf4fd8899c1c1931251308 P 51386e0b5b304a50b1e7fc6a8a7d6447 [term 1 FOLLOWER]: Refusing update from remote peer 4e7ad3026ef0490d8f87163cf83288c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:46:51.144268 12804 consensus_queue.cc:1035] T fa4f13de5aaf4fd8899c1c1931251308 P 4e7ad3026ef0490d8f87163cf83288c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dde73add689c439597720e3fafceb342" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 43887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:46:51.144662 12800 consensus_queue.cc:1035] T fa4f13de5aaf4fd8899c1c1931251308 P 4e7ad3026ef0490d8f87163cf83288c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51386e0b5b304a50b1e7fc6a8a7d6447" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 39313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:46:51.156113 12807 mvcc.cc:205] Tried to move back new op lower bound from 7315392148031242240 to 7315392147850178560. Current Snapshot: MvccSnapshot[applied={T|T < 7315392148031242240}]
I20260806 02:46:51.181968 12316 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:46:51.182331 12316 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:46:51.186774 12316 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 548c70043c2348cb9373c187ffd716bd: Sending DeleteTablet for 3 replicas of tablet fa4f13de5aaf4fd8899c1c1931251308
I20260806 02:46:51.188190 12576 tablet_service.cc:1514] Processing DeleteTablet for tablet fa4f13de5aaf4fd8899c1c1931251308 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:46:51 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:59762
I20260806 02:46:51.188263 12441 tablet_service.cc:1514] Processing DeleteTablet for tablet fa4f13de5aaf4fd8899c1c1931251308 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:46:51 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:39256
I20260806 02:46:51.188766 12710 tablet_service.cc:1514] Processing DeleteTablet for tablet fa4f13de5aaf4fd8899c1c1931251308 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:46:51 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:60580
I20260806 02:46:51.189540 12818 tablet_replica.cc:330] T fa4f13de5aaf4fd8899c1c1931251308 P dde73add689c439597720e3fafceb342: stopping tablet replica
I20260806 02:46:51.190121 12818 raft_consensus.cc:2229] T fa4f13de5aaf4fd8899c1c1931251308 P dde73add689c439597720e3fafceb342 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:46:51.190327 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 12378
I20260806 02:46:51.190805 12818 raft_consensus.cc:2258] T fa4f13de5aaf4fd8899c1c1931251308 P dde73add689c439597720e3fafceb342 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:46:51.192003 12819 tablet_replica.cc:330] T fa4f13de5aaf4fd8899c1c1931251308 P 4e7ad3026ef0490d8f87163cf83288c1: stopping tablet replica
I20260806 02:46:51.192641 12819 raft_consensus.cc:2229] T fa4f13de5aaf4fd8899c1c1931251308 P 4e7ad3026ef0490d8f87163cf83288c1 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:46:51.193565 12819 raft_consensus.cc:2258] T fa4f13de5aaf4fd8899c1c1931251308 P 4e7ad3026ef0490d8f87163cf83288c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:46:51.196748 12818 ts_tablet_manager.cc:1905] T fa4f13de5aaf4fd8899c1c1931251308 P dde73add689c439597720e3fafceb342: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:46:51.201356 12817 tablet_replica.cc:330] T fa4f13de5aaf4fd8899c1c1931251308 P 51386e0b5b304a50b1e7fc6a8a7d6447: stopping tablet replica
I20260806 02:46:51.202059 12817 raft_consensus.cc:2229] T fa4f13de5aaf4fd8899c1c1931251308 P 51386e0b5b304a50b1e7fc6a8a7d6447 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:46:51.202780 12817 raft_consensus.cc:2258] T fa4f13de5aaf4fd8899c1c1931251308 P 51386e0b5b304a50b1e7fc6a8a7d6447 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:46:51.205358 12817 ts_tablet_manager.cc:1905] T fa4f13de5aaf4fd8899c1c1931251308 P 51386e0b5b304a50b1e7fc6a8a7d6447: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:46:51.205390 12819 ts_tablet_manager.cc:1905] T fa4f13de5aaf4fd8899c1c1931251308 P 4e7ad3026ef0490d8f87163cf83288c1: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 06 02:46:51 dist-test-slave-8tdl krb5kdc[12272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.65@KRBTEST.COM
I20260806 02:46:51.211148 12818 ts_tablet_manager.cc:1918] T fa4f13de5aaf4fd8899c1c1931251308 P dde73add689c439597720e3fafceb342: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:46:51.211387 12818 log.cc:1196] T fa4f13de5aaf4fd8899c1c1931251308 P dde73add689c439597720e3fafceb342: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-0/wal/wals/fa4f13de5aaf4fd8899c1c1931251308
I20260806 02:46:51.213856 12817 ts_tablet_manager.cc:1918] T fa4f13de5aaf4fd8899c1c1931251308 P 51386e0b5b304a50b1e7fc6a8a7d6447: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:46:51.214180 12817 log.cc:1196] T fa4f13de5aaf4fd8899c1c1931251308 P 51386e0b5b304a50b1e7fc6a8a7d6447: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-1/wal/wals/fa4f13de5aaf4fd8899c1c1931251308
I20260806 02:46:51.214890 12817 ts_tablet_manager.cc:1939] T fa4f13de5aaf4fd8899c1c1931251308 P 51386e0b5b304a50b1e7fc6a8a7d6447: Deleting consensus metadata
I20260806 02:46:51.212028 12818 ts_tablet_manager.cc:1939] T fa4f13de5aaf4fd8899c1c1931251308 P dde73add689c439597720e3fafceb342: Deleting consensus metadata
I20260806 02:46:51.217021 12819 ts_tablet_manager.cc:1918] T fa4f13de5aaf4fd8899c1c1931251308 P 4e7ad3026ef0490d8f87163cf83288c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:46:51.217231 12819 log.cc:1196] T fa4f13de5aaf4fd8899c1c1931251308 P 4e7ad3026ef0490d8f87163cf83288c1: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984304136196-1865-0/minicluster-data/ts-2/wal/wals/fa4f13de5aaf4fd8899c1c1931251308
I20260806 02:46:51.217258 12303 catalog_manager.cc:4841] TS dde73add689c439597720e3fafceb342 (127.1.210.65:43887): tablet fa4f13de5aaf4fd8899c1c1931251308 (table test-table [id=f7cbc640ce6f434d804ed1770a779dca]) successfully deleted
I20260806 02:46:51.217330 12304 catalog_manager.cc:4841] TS 51386e0b5b304a50b1e7fc6a8a7d6447 (127.1.210.66:39313): tablet fa4f13de5aaf4fd8899c1c1931251308 (table test-table [id=f7cbc640ce6f434d804ed1770a779dca]) successfully deleted
I20260806 02:46:51.217887 12819 ts_tablet_manager.cc:1939] T fa4f13de5aaf4fd8899c1c1931251308 P 4e7ad3026ef0490d8f87163cf83288c1: Deleting consensus metadata
I20260806 02:46:51.219823 12301 catalog_manager.cc:4841] TS 4e7ad3026ef0490d8f87163cf83288c1 (127.1.210.67:46449): tablet fa4f13de5aaf4fd8899c1c1931251308 (table test-table [id=f7cbc640ce6f434d804ed1770a779dca]) successfully deleted
I20260806 02:46:51.220395 12792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:51.193820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39272 (local address 127.1.210.65:43887)
0806 02:46:51.193997 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:51.194004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:51.194581 (+ 577us) server_negotiation.cc:407] Connection header received
0806 02:46:51.194997 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:51.195007 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:51.195125 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:51.195240 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:51.197585 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:51.199286 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:51.202002 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:51.202733 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:51.202875 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:51.209170 (+ 6295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:51.209193 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:51.209211 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:51.209268 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:51.213995 (+ 4727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:51.217675 (+ 3680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:51.217686 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:51.217689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:51.217894 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:51.218498 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:51.218505 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:51.218507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:51.219025 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:51.219430 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:51.219863 (+ 433us) server_negotiation.cc:299] Negotiation successful
0806 02:46:51.220125 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:46:51.221884 12501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:46:51.314899 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 12378
I20260806 02:46:51.335829 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 12512
Aug 06 02:46:51 dist-test-slave-8tdl krb5kdc[12272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.66@KRBTEST.COM
I20260806 02:46:51.355208 12789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:51.336816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59770 (local address 127.1.210.66:39313)
0806 02:46:51.337051 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:51.337058 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:51.337088 (+ 30us) server_negotiation.cc:407] Connection header received
0806 02:46:51.337226 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:51.337232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:51.337327 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:51.337459 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:51.339035 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:51.339818 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:51.341285 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:51.341747 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:51.341972 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:51.346364 (+ 4392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:51.346398 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:51.346422 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:51.346474 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:51.350842 (+ 4368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:51.351826 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:51.351837 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:51.351842 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:51.352024 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:51.352603 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:51.352610 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:51.352613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:51.353519 (+ 906us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:51.353884 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:51.354345 (+ 461us) server_negotiation.cc:299] Negotiation successful
0806 02:46:51.354635 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 02:46:51.356155 12636 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:46:51.450670 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 12512
I20260806 02:46:51.470157 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 12647
Aug 06 02:46:51 dist-test-slave-8tdl krb5kdc[12272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.67@KRBTEST.COM
I20260806 02:46:51.488422 12791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:51.471010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60600 (local address 127.1.210.67:46449)
0806 02:46:51.471223 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:51.471229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:51.471306 (+ 77us) server_negotiation.cc:407] Connection header received
0806 02:46:51.471532 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:51.471538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:51.471642 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:51.471764 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:51.473315 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:51.474193 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:51.475677 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:51.476163 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:51.476381 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:51.481205 (+ 4824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:51.481223 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:51.481242 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:51.481311 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:51.485046 (+ 3735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:51.485931 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:51.485940 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:51.485942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:51.486148 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:51.486579 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:51.486591 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:51.486594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:51.487039 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:51.487399 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:51.487749 (+ 350us) server_negotiation.cc:299] Negotiation successful
0806 02:46:51.487992 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260806 02:46:51.489120 12770 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:46:51.620517 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 12647
I20260806 02:46:51.642405 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 12285
I20260806 02:46:51.643731 12347 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:46:51.761461 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 12285
2026-08-06T02:46:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4836 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:46:51 dist-test-slave-8tdl krb5kdc[12827](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:46:51 dist-test-slave-8tdl krb5kdc[12827](info): set up 2 sockets
Aug 06 02:46:51 dist-test-slave-8tdl krb5kdc[12827](info): commencing operation
krb5kdc: starting...
W20260806 02:46:52.910857 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.005s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:46:53 dist-test-slave-8tdl krb5kdc[12827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:46:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:46:53Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.210.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.210.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.210.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/rangeradmin_127.1.210.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.210.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/rangeradmin_127.1.210.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.210.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.210.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.210.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/rangerlookup_127.1.210.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.210.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/rangerlookup_127.1.210.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.210.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.210.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.210.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/HTTP_127.1.210.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.210.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/HTTP_127.1.210.84@KRBTEST.COM.keytab.
I20260806 02:46:53.326575 1865 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task5Fb5fs/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/postgres -l logfile start
2026-08-06 02:46:56.564 UTC [12857] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 02:46:56.564 UTC [12857] LOG: listening on IPv4 address "127.1.210.84", port 43333
2026-08-06 02:46:56.573 UTC [12857] LOG: listening on Unix socket "/tmp/.s.PGSQL.43333"
2026-08-06 02:46:56.597 UTC [12860] LOG: database system was shut down at 2026-08-06 02:46:54 UTC
2026-08-06 02:46:56.601 UTC [12857] LOG: database system is ready to accept connections
I20260806 02:46:57.618804 1865 mini_postgres.cc:96] Postgres bound to 43333
2026-08-06 02:46:57.642 UTC [12868] FATAL: database "slave" does not exist
127.1.210.84:43333 - accepting connections
I20260806 02:46:57.643517 1865 mini_ranger.cc:161] Starting Ranger...
I20260806 02:46:57.677052 1865 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 02:46:58.346469 1865 mini_ranger.cc:87] Created ranger Postgres database
I20260806 02:46:58.346628 1865 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ranger-admin
2026-08-06 02:46:58,695 [I] DB FLAVOR :POSTGRES
2026-08-06 02:46:58,695 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 02:46:58,696 [I] Checking connection..
2026-08-06 02:46:58,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:43333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 02:46:58,991 [I] Checking connection passed.
2026-08-06 02:46:58,991 [I] --------- Verifying version history table ---------
2026-08-06 02:46:58,991 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:43333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 02:46:59,271 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 02:46:59,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:43333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 02:46:59,567 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 02:46:59,567 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 02:46:59,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:43333/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 02:46:59.840 UTC [12932] WARNING: there is no transaction in progress
2026-08-06 02:46:59,848 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 02:46:59,848 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:43333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 02:47:00,149 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 02:47:00,149 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 02:47:00,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:43333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 02:47:00,439 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:43333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 02:47:00,697 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:43333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8tdl.c.gcp-upstream.internal','N') ;"
2026-08-06 02:47:00,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:43333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 02:47:01,241 [I] Table x_portal_user does not exist in database ranger
2026-08-06 02:47:01,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:43333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 02:47:01,530 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 02:47:01,531 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:43333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 02:47:01,793 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:43333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 02:47:02,077 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 02:47:02,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:43333/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 02:47:03.372 UTC [13040] WARNING: there is no transaction in progress
2026-08-06 02:47:03.393 UTC [13040] WARNING: there is no transaction in progress
2026-08-06 02:47:03.415 UTC [13040] WARNING: there is no transaction in progress
2026-08-06 02:47:03.436 UTC [13040] WARNING: there is no transaction in progress
2026-08-06 02:47:03.459 UTC [13040] WARNING: there is no transaction in progress
2026-08-06 02:47:03.487 UTC [13040] WARNING: there is no transaction in progress
2026-08-06 02:47:03.570 UTC [13040] WARNING: there is no transaction in progress
2026-08-06 02:47:03.580 UTC [13040] WARNING: there is no transaction in progress
2026-08-06 02:47:03.604 UTC [13040] WARNING: there is no transaction in progress
2026-08-06 02:47:03.619 UTC [13040] WARNING: there is no transaction in progress
2026-08-06 02:47:03.634 UTC [13040] WARNING: there is no transaction in progress
2026-08-06 02:47:03.657 UTC [13040] WARNING: there is no transaction in progress
2026-08-06 02:47:03.671 UTC [13040] WARNING: there is no transaction in progress
2026-08-06 02:47:03.680 UTC [13040] WARNING: there is no transaction in progress
2026-08-06 02:47:03.691 UTC [13040] WARNING: there is no transaction in progress
2026-08-06 02:47:03.701 UTC [13040] WARNING: there is no transaction in progress
2026-08-06 02:47:04,707 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 02:47:04,707 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:43333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 02:47:04,949 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 02:47:04,949 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:43333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 02:47:05,232 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 02:47:05,232 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:43333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 02:47:05,515 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 02:47:05,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:43333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 02:47:05,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:43333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 02:47:06,055 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:43333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8tdl.c.gcp-upstream.internal';"
2026-08-06 02:47:06,312 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 02:47:06,313 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:43333/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 02:47:06,592 [I] DB_PATCHES have already been applied
I20260806 02:47:06.598547 1865 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ranger-admin:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task5Fb5fs/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 02:47:06.598649 1865 mini_ranger.cc:193] Using host: 127.1.210.84
I20260806 02:47:06.619589 1865 mini_ranger.cc:227] Ranger admin URL: http://127.1.210.84:58503
Aug 06, 2026 2:47:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 2:47:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 2:47:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 2:47:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 2:47:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 2:47:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 2:47:07 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 02:47:07.851289 1865 mini_ranger.cc:160] Time spent starting Ranger: real 10.208s user 0.012s sys 0.052s
Aug 06, 2026 2:47:29 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 2:47:29 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 06, 2026 2:47:29 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 2:47:29 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 2:47:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 2:47:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 2:47:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 2:47:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 2:47:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 2:47:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 2:47:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 2:47:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 2:47:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 2:47:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 2:47:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 2:47:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 2:47:29 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 2:47:30 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 02:47:30.963400 1865 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.210.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.210.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.210.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/rangerkms_127.1.210.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.210.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/rangerkms_127.1.210.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.210.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.210.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.210.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/HTTP_127.1.210.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.210.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/HTTP_127.1.210.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 02:47:31 dist-test-slave-8tdl krb5kdc[12827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984451, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 02:47:31.174772 1865 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 02:47:31.202298 1865 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 02:47:32.272245 1865 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 02:47:32.272440 1865 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ranger-kms
I20260806 02:47:32.272608 1865 mini_ranger_kms.cc:227] Using postgres at 127.1.210.84:43333
2026-08-06 02:47:32,896 [I] DB FLAVOR :POSTGRES
2026-08-06 02:47:32,896 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 02:47:32,896 [I] Checking connection
2026-08-06 02:47:32,896 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:43333/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 02:47:33,165 [I] connection success
2026-08-06 02:47:33,165 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 02:47:33,165 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 02:47:33,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:43333/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 02:47:33,421 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 02:47:33,421 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 02:47:33,421 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 02:47:33,421 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:43333/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 02:47:33,701 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 02:47:34.225853 1865 mini_ranger_kms.cc:310] Created kms service
I20260806 02:47:34.442135 1865 mini_ranger_kms.cc:326] Created kudu user
I20260806 02:47:34.495162 1865 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 02:47:34 dist-test-slave-8tdl krb5kdc[12827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984451, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.210.84@KRBTEST.COM
I20260806 02:47:34.907778 1865 mini_ranger_kms.cc:384] Added ranger policy
I20260806 02:47:34.909176 1865 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ranger-kms:/tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 02:47:34.909257 1865 mini_ranger_kms.cc:240] Using host: 127.1.210.84
I20260806 02:47:34.938865 1865 mini_ranger_kms.cc:276] Ranger KMS PID: 13221
I20260806 02:47:34.939064 1865 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.210.84:53127
Aug 06, 2026 2:47:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 2:47:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 2:47:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 2:47:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 2:47:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 2:47:36 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260806 02:47:37.285718 1865 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.111s user 0.162s sys 0.489s
I20260806 02:47:37.286058 1865 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 02:47:37.286295 1865 mini_ranger_kms.cc:401] 127.1.210.84:53127/kms/v1/keys
Aug 06 02:47:37 dist-test-slave-8tdl krb5kdc[12827](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785984457, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.210.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 06 02:47:37 dist-test-slave-8tdl krb5kdc[12827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984457, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.210.84@KRBTEST.COM for HTTP/127.1.210.84@KRBTEST.COM
Aug 06 02:47:38 dist-test-slave-8tdl krb5kdc[12827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984457, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.210.84@KRBTEST.COM for HTTP/127.1.210.84@KRBTEST.COM
Aug 06 02:47:40 dist-test-slave-8tdl krb5kdc[12827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
I20260806 02:47:41.017845 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:34347
--webserver_interface=127.1.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:38245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:34347
--ranger_config_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.210.84:53127/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:47:41.759827 13271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:47:41.760324 13271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:47:41.760644 13271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:47:41.781273 13271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:47:41.781553 13271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:47:41.781737 13271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:47:41.781952 13271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:47:41.782126 13271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:47:41.798461 13271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:38245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.210.84:53127/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:34347
--ranger_config_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:34347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:47:41.800097 13271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:47:41.801537 13271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:47:41.817410 13276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:47:41.817579 13277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:47:41.817603 13279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:47:41.818205 13271 server_base.cc:1034] running on GCE node
I20260806 02:47:41.819303 13271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:47:41.821161 13271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:47:41.822415 13271 hybrid_clock.cc:648] HybridClock initialized: now 1785984461822349 us; error 70 us; skew 500 ppm
I20260806 02:47:41.823102 13271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:47:41 dist-test-slave-8tdl krb5kdc[12827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984461, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:47:41.826998 13271 init.cc:377] Logged in from keytab as kudu/127.1.210.126@KRBTEST.COM (short username kudu)
I20260806 02:47:41.829046 13271 webserver.cc:460] Webserver started at http://127.1.210.126:33851/ using document root <none> and password file <none>
I20260806 02:47:41.829938 13271 fs_manager.cc:359] Metadata directory not provided
I20260806 02:47:41.830170 13271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:47:41.830832 13271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:47:41 dist-test-slave-8tdl krb5kdc[12827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984461, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for HTTP/127.1.210.84@KRBTEST.COM
I20260806 02:47:41.901093 13271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "0f387ce1ddc944ec973e0eac647a9986"
format_stamp: "Formatted at 2026-08-06 02:47:41 on dist-test-slave-8tdl"
server_key: "e0d117806a96151b0fe9e2771172eac7"
server_key_iv: "f58c9534b60e1183da63192743c7f9e3"
server_key_version: "kuduclusterkey@0"
I20260806 02:47:41.902164 13271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "0f387ce1ddc944ec973e0eac647a9986"
format_stamp: "Formatted at 2026-08-06 02:47:41 on dist-test-slave-8tdl"
server_key: "e0d117806a96151b0fe9e2771172eac7"
server_key_iv: "f58c9534b60e1183da63192743c7f9e3"
server_key_version: "kuduclusterkey@0"
I20260806 02:47:41.909879 13271 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260806 02:47:41.931583 13286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:47:41.933373 13271 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260806 02:47:41.933657 13271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "0f387ce1ddc944ec973e0eac647a9986"
format_stamp: "Formatted at 2026-08-06 02:47:41 on dist-test-slave-8tdl"
server_key: "e0d117806a96151b0fe9e2771172eac7"
server_key_iv: "f58c9534b60e1183da63192743c7f9e3"
server_key_version: "kuduclusterkey@0"
I20260806 02:47:41.933933 13271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:47:41.960983 13271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:47:41.967777 13271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:47:41.968420 13271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:47:42.022082 13271 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:34347
I20260806 02:47:42.022292 13337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:34347 every 8 connection(s)
I20260806 02:47:42.024194 13271 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:47:42.025821 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 13271
I20260806 02:47:42.026136 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:47:42.027526 1865 external_mini_cluster.cc:1428] Setting key cafb3daa40bc3f3125c3c85d3b58c0ed
I20260806 02:47:42.032124 13338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 02:47:42 dist-test-slave-8tdl krb5kdc[12827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:47:42.056013 13338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f387ce1ddc944ec973e0eac647a9986: Bootstrap starting.
I20260806 02:47:42.063882 13338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f387ce1ddc944ec973e0eac647a9986: Neither blocks nor log segments found. Creating new log.
I20260806 02:47:42.066812 13338 log.cc:824] T 00000000000000000000000000000000 P 0f387ce1ddc944ec973e0eac647a9986: Log is configured to *not* fsync() on all Append() calls
I20260806 02:47:42.067397 13341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:42.031134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59498 (local address 127.1.210.126:34347)
0806 02:47:42.033396 (+ 2262us) server_negotiation.cc:206] Beginning negotiation
0806 02:47:42.033409 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:47:42.033474 (+ 65us) server_negotiation.cc:407] Connection header received
0806 02:47:42.036388 (+ 2914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:47:42.036415 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:47:42.036907 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:47:42.037357 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:47:42.039726 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:42.042555 (+ 2829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:42.044800 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:42.045853 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:42.046006 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:42.052393 (+ 6387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:47:42.052410 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:47:42.052436 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:47:42.052539 (+ 103us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:47:42.060122 (+ 7583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:47:42.061303 (+ 1181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:47:42.061315 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:47:42.061325 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:47:42.061647 (+ 322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:47:42.062161 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:47:42.062166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:47:42.062169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:47:42.063367 (+ 1198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:47:42.063851 (+ 484us) server_negotiation.cc:1035] Waiting for connection context
0806 02:47:42.064295 (+ 444us) server_negotiation.cc:299] Negotiation successful
0806 02:47:42.065123 (+ 828us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1917,"thread_start_us":1465,"threads_started":1}
I20260806 02:47:42.073917 13338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f387ce1ddc944ec973e0eac647a9986: No bootstrap required, opened a new log
I20260806 02:47:42.086964 13338 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f387ce1ddc944ec973e0eac647a9986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f387ce1ddc944ec973e0eac647a9986" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 34347 } }
I20260806 02:47:42.087517 13338 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f387ce1ddc944ec973e0eac647a9986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:47:42.087688 13338 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f387ce1ddc944ec973e0eac647a9986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f387ce1ddc944ec973e0eac647a9986, State: Initialized, Role: FOLLOWER
I20260806 02:47:42.088522 13338 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f387ce1ddc944ec973e0eac647a9986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f387ce1ddc944ec973e0eac647a9986" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 34347 } }
I20260806 02:47:42.088837 13338 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f387ce1ddc944ec973e0eac647a9986 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:47:42.089036 13338 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f387ce1ddc944ec973e0eac647a9986 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:47:42.089269 13338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f387ce1ddc944ec973e0eac647a9986 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:47:42.092413 13338 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f387ce1ddc944ec973e0eac647a9986 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f387ce1ddc944ec973e0eac647a9986" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 34347 } }
I20260806 02:47:42.093065 13338 leader_election.cc:302] T 00000000000000000000000000000000 P 0f387ce1ddc944ec973e0eac647a9986 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f387ce1ddc944ec973e0eac647a9986; no voters:
I20260806 02:47:42.094281 13338 leader_election.cc:288] T 00000000000000000000000000000000 P 0f387ce1ddc944ec973e0eac647a9986 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:47:42.094339 13343 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f387ce1ddc944ec973e0eac647a9986 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:47:42.096568 13343 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f387ce1ddc944ec973e0eac647a9986 [term 1 LEADER]: Becoming Leader. State: Replica: 0f387ce1ddc944ec973e0eac647a9986, State: Running, Role: LEADER
I20260806 02:47:42.097465 13343 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f387ce1ddc944ec973e0eac647a9986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f387ce1ddc944ec973e0eac647a9986" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 34347 } }
I20260806 02:47:42.098568 13338 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f387ce1ddc944ec973e0eac647a9986 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:47:42.102109 13344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f387ce1ddc944ec973e0eac647a9986 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f387ce1ddc944ec973e0eac647a9986. Latest consensus state: current_term: 1 leader_uuid: "0f387ce1ddc944ec973e0eac647a9986" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f387ce1ddc944ec973e0eac647a9986" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 34347 } } }
I20260806 02:47:42.103299 13344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f387ce1ddc944ec973e0eac647a9986 [sys.catalog]: This master's current role is: LEADER
I20260806 02:47:42.104234 13345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f387ce1ddc944ec973e0eac647a9986 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f387ce1ddc944ec973e0eac647a9986" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f387ce1ddc944ec973e0eac647a9986" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 34347 } } }
I20260806 02:47:42.105010 13345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f387ce1ddc944ec973e0eac647a9986 [sys.catalog]: This master's current role is: LEADER
I20260806 02:47:42.112263 13352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:47:42.119696 13352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:47:42.120970 13338 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 02:47:42.143790 13352 catalog_manager.cc:1347] Generated new cluster ID: afb986515a6b494eadbf275fab63a6fa
I20260806 02:47:42.143982 13352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:47:42.176856 13352 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:47:42.179499 13352 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:47:42.215133 13352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f387ce1ddc944ec973e0eac647a9986: Generated new TSK 0
I20260806 02:47:42.216403 13352 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab.
Entry for principal kudu/127.1.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab.
I20260806 02:47:43.120141 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.210.84:53127/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.65:0
--local_ip_for_outbound_sockets=127.1.210.65
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:34347
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:38245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:47:43 dist-test-slave-8tdl krb5kdc[12827](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785984463, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 02:47:43.679764 13383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:47:43.680295 13383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:47:43.680583 13383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:47:43.691380 13383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:47:43.691709 13383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:47:43.692109 13383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.65
I20260806 02:47:43.707131 13383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:38245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.210.84:53127/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:34347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.65
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:47:43.708869 13383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:47:43.710757 13383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:47:43.745230 13394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:47:43.746381 13393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:47:43.751643 13383 server_base.cc:1034] running on GCE node
W20260806 02:47:43.752807 13397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:47:43.753995 13383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:47:43.755909 13383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:47:43.757332 13383 hybrid_clock.cc:648] HybridClock initialized: now 1785984463757171 us; error 74 us; skew 500 ppm
I20260806 02:47:43.758064 13383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:47:43 dist-test-slave-8tdl krb5kdc[12827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984463, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:47:43.764135 13383 init.cc:377] Logged in from keytab as kudu/127.1.210.65@KRBTEST.COM (short username kudu)
I20260806 02:47:43.769343 13383 webserver.cc:460] Webserver started at http://127.1.210.65:36075/ using document root <none> and password file <none>
I20260806 02:47:43.770350 13383 fs_manager.cc:359] Metadata directory not provided
I20260806 02:47:43.770641 13383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:47:43.771395 13383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:47:43 dist-test-slave-8tdl krb5kdc[12827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984463, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for HTTP/127.1.210.84@KRBTEST.COM
I20260806 02:47:43.794955 13383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-0/data/instance:
uuid: "8129982a4d914f71b531306b72dacab0"
format_stamp: "Formatted at 2026-08-06 02:47:43 on dist-test-slave-8tdl"
server_key: "68a415e968177cd1551e7cb651127648"
server_key_iv: "8ce8e3398810891829ccf64b26a06f9d"
server_key_version: "kuduclusterkey@0"
I20260806 02:47:43.796131 13383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance:
uuid: "8129982a4d914f71b531306b72dacab0"
format_stamp: "Formatted at 2026-08-06 02:47:43 on dist-test-slave-8tdl"
server_key: "68a415e968177cd1551e7cb651127648"
server_key_iv: "8ce8e3398810891829ccf64b26a06f9d"
server_key_version: "kuduclusterkey@0"
I20260806 02:47:43.805109 13383 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.008s sys 0.001s
I20260806 02:47:43.827461 13403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:47:43.829602 13383 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.003s
I20260806 02:47:43.829874 13383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-0/wal
uuid: "8129982a4d914f71b531306b72dacab0"
format_stamp: "Formatted at 2026-08-06 02:47:43 on dist-test-slave-8tdl"
server_key: "68a415e968177cd1551e7cb651127648"
server_key_iv: "8ce8e3398810891829ccf64b26a06f9d"
server_key_version: "kuduclusterkey@0"
I20260806 02:47:43.830665 13383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:47:43.872807 13383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:47:43.880726 13383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:47:43.881371 13383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:47:43.883993 13383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:47:43.887364 13383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:47:43.887598 13383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:47:43.887929 13383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:47:43.888095 13383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:47:44.078341 13383 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.65:46093
I20260806 02:47:44.078629 13515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.65:46093 every 8 connection(s)
I20260806 02:47:44.080325 13383 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
I20260806 02:47:44.088722 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 13383
I20260806 02:47:44.089076 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance
I20260806 02:47:44.090397 1865 external_mini_cluster.cc:1428] Setting key 428e3fc3423d56fb7f34569c7b385c62
Aug 06 02:47:44 dist-test-slave-8tdl krb5kdc[12827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984463, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for HTTP/127.1.210.84@KRBTEST.COM
Aug 06 02:47:44 dist-test-slave-8tdl krb5kdc[12827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984463, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.210.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.66@KRBTEST.COM" created.
I20260806 02:47:44.168216 13518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:44.085051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:45279 (local address 127.1.210.126:34347)
0806 02:47:44.131848 (+ 46797us) server_negotiation.cc:206] Beginning negotiation
0806 02:47:44.131856 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:47:44.131888 (+ 32us) server_negotiation.cc:407] Connection header received
0806 02:47:44.133895 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:47:44.133904 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:47:44.134038 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:47:44.134167 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:47:44.140442 (+ 6275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:44.142247 (+ 1805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:44.146052 (+ 3805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:44.147112 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:44.147232 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:44.155856 (+ 8624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:47:44.155866 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:47:44.155872 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:47:44.155924 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:47:44.162768 (+ 6844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:47:44.164326 (+ 1558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:47:44.164334 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:47:44.164337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:47:44.164586 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:47:44.165698 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:47:44.165704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:47:44.165707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:47:44.166157 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:47:44.166302 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 02:47:44.167393 (+ 1091us) server_negotiation.cc:299] Negotiation successful
0806 02:47:44.167654 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46579,"thread_start_us":49617,"threads_started":1}
I20260806 02:47:44.171264 13519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:44.086064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:34347 (local address 127.1.210.65:45279)
0806 02:47:44.119666 (+ 33602us) negotiation.cc:107] Waiting for socket to connect
0806 02:47:44.119776 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0806 02:47:44.133233 (+ 13457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:47:44.134962 (+ 1729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:47:44.134982 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:47:44.136026 (+ 1044us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:47:44.140089 (+ 4063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:47:44.140133 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:47:44.142635 (+ 2502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:47:44.142641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:47:44.145795 (+ 3154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:47:44.145815 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:47:44.147794 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:47:44.147805 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:47:44.148252 (+ 447us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:44.150801 (+ 2549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:47:44.150856 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:47:44.155426 (+ 4570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:47:44.163028 (+ 7602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:47:44.163040 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:47:44.163064 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:47:44.163981 (+ 917us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:47:44.164797 (+ 816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:47:44.164802 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:47:44.164806 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:47:44.165540 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:47:44.166407 (+ 867us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:47:44.166416 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:47:44.166940 (+ 524us) client_negotiation.cc:769] Sending connection context
0806 02:47:44.168515 (+ 1575us) client_negotiation.cc:240] Negotiation successful
0806 02:47:44.169359 (+ 844us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33047,"thread_start_us":36461,"threads_started":1}
I20260806 02:47:44.173461 13516 heartbeater.cc:347] Connected to a master server at 127.1.210.126:34347
I20260806 02:47:44.173841 13516 heartbeater.cc:464] Registering TS with master...
I20260806 02:47:44.177946 13516 heartbeater.cc:511] Master 127.1.210.126:34347 requested a full tablet report, sending...
I20260806 02:47:44.181656 13303 ts_manager.cc:194] Registered new tserver with Master: 8129982a4d914f71b531306b72dacab0 (127.1.210.65:46093)
I20260806 02:47:44.184613 13303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.65@KRBTEST.COM'} at 127.1.210.65:45279
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab.
Entry for principal kudu/127.1.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab.
W20260806 02:47:44.244179 1865 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.1.210.66: real 0.150s user 0.000s sys 0.022s
I20260806 02:47:44.245489 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.210.84:53127/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.66:0
--local_ip_for_outbound_sockets=127.1.210.66
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:34347
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:38245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:47:44 dist-test-slave-8tdl krb5kdc[12827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984463, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for HTTP/127.1.210.84@KRBTEST.COM
W20260806 02:47:45.126076 13522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:47:45.126518 13522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:47:45.126770 13522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:47:45.139194 13522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:47:45.139479 13522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:47:45.139775 13522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.66
I20260806 02:47:45.154140 13522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:38245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.210.84:53127/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:34347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.66
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:47:45.155864 13522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:47:45.157510 13522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:47:45.179847 13536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:47:45.180349 13537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:47:45.180733 13539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:47:45.182292 13522 server_base.cc:1034] running on GCE node
I20260806 02:47:45.183094 13522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:47:45.184237 13522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:47:45.185585 13522 hybrid_clock.cc:648] HybridClock initialized: now 1785984465185475 us; error 80 us; skew 500 ppm
I20260806 02:47:45.186168 13522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:47:45.190117 13516 heartbeater.cc:503] Master 127.1.210.126:34347 was elected leader, sending a full tablet report...
Aug 06 02:47:45 dist-test-slave-8tdl krb5kdc[12827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984465, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:47:45.191855 13522 init.cc:377] Logged in from keytab as kudu/127.1.210.66@KRBTEST.COM (short username kudu)
I20260806 02:47:45.194594 13522 webserver.cc:460] Webserver started at http://127.1.210.66:38285/ using document root <none> and password file <none>
I20260806 02:47:45.195628 13522 fs_manager.cc:359] Metadata directory not provided
I20260806 02:47:45.195889 13522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:47:45.196413 13522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:47:45 dist-test-slave-8tdl krb5kdc[12827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984465, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for HTTP/127.1.210.84@KRBTEST.COM
I20260806 02:47:45.214437 13522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-1/data/instance:
uuid: "89701522673d4020993463f9228b73ee"
format_stamp: "Formatted at 2026-08-06 02:47:45 on dist-test-slave-8tdl"
server_key: "44a89cb34d26d7ab7913de962f258170"
server_key_iv: "cf39fc6a09f6c7a04014df372850776c"
server_key_version: "kuduclusterkey@0"
I20260806 02:47:45.215695 13522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance:
uuid: "89701522673d4020993463f9228b73ee"
format_stamp: "Formatted at 2026-08-06 02:47:45 on dist-test-slave-8tdl"
server_key: "44a89cb34d26d7ab7913de962f258170"
server_key_iv: "cf39fc6a09f6c7a04014df372850776c"
server_key_version: "kuduclusterkey@0"
I20260806 02:47:45.223815 13522 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260806 02:47:45.245461 13546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:47:45.247393 13522 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260806 02:47:45.247663 13522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-1/wal
uuid: "89701522673d4020993463f9228b73ee"
format_stamp: "Formatted at 2026-08-06 02:47:45 on dist-test-slave-8tdl"
server_key: "44a89cb34d26d7ab7913de962f258170"
server_key_iv: "cf39fc6a09f6c7a04014df372850776c"
server_key_version: "kuduclusterkey@0"
I20260806 02:47:45.247962 13522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:47:45.280403 13522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:47:45.285983 13522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:47:45.286515 13522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:47:45.288282 13522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:47:45.290506 13522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:47:45.290635 13522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:47:45.290930 13522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:47:45.291006 13522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:47:45.371387 13522 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.66:45753
I20260806 02:47:45.371460 13658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.66:45753 every 8 connection(s)
I20260806 02:47:45.373152 13522 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
I20260806 02:47:45.381311 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 13522
I20260806 02:47:45.381604 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance
I20260806 02:47:45.382280 1865 external_mini_cluster.cc:1428] Setting key 6e82b699670cfd815339f4bc050fab5a
Aug 06 02:47:45 dist-test-slave-8tdl krb5kdc[12827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984465, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:47:45.409312 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:45.377275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:36781 (local address 127.1.210.126:34347)
0806 02:47:45.378261 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0806 02:47:45.378268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:47:45.379911 (+ 1643us) server_negotiation.cc:407] Connection header received
0806 02:47:45.383535 (+ 3624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:47:45.383541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:47:45.383638 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:47:45.383778 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:47:45.386771 (+ 2993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:45.387872 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:45.391594 (+ 3722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:45.392216 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:45.392367 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:45.398572 (+ 6205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:47:45.398582 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:47:45.398590 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:47:45.398722 (+ 132us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:47:45.404645 (+ 5923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:47:45.405796 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:47:45.405801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:47:45.405804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:47:45.406056 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:47:45.406707 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:47:45.406714 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:47:45.406718 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:47:45.407221 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:47:45.407405 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0806 02:47:45.408344 (+ 939us) server_negotiation.cc:299] Negotiation successful
0806 02:47:45.408613 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":776,"thread_start_us":555,"threads_started":1}
I20260806 02:47:45.410629 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:45.378093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:34347 (local address 127.1.210.66:36781)
0806 02:47:45.379674 (+ 1581us) negotiation.cc:107] Waiting for socket to connect
0806 02:47:45.379716 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:47:45.383189 (+ 3473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:47:45.384131 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:47:45.384141 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:47:45.385084 (+ 943us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:47:45.386474 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:47:45.386507 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:47:45.388044 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:47:45.388052 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:47:45.391405 (+ 3353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:47:45.391427 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:47:45.392614 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:47:45.392626 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:47:45.393091 (+ 465us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:45.394080 (+ 989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:47:45.394116 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:47:45.398270 (+ 4154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:47:45.404860 (+ 6590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:47:45.404870 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:47:45.404918 (+ 48us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:47:45.405601 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:47:45.406196 (+ 595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:47:45.406202 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:47:45.406206 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:47:45.406546 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:47:45.407414 (+ 868us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:47:45.407426 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:47:45.407840 (+ 414us) client_negotiation.cc:769] Sending connection context
0806 02:47:45.408158 (+ 318us) client_negotiation.cc:240] Negotiation successful
0806 02:47:45.408981 (+ 823us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1229,"thread_start_us":828,"threads_started":1}
I20260806 02:47:45.412623 13659 heartbeater.cc:347] Connected to a master server at 127.1.210.126:34347
I20260806 02:47:45.413127 13659 heartbeater.cc:464] Registering TS with master...
I20260806 02:47:45.414480 13659 heartbeater.cc:511] Master 127.1.210.126:34347 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.210.67@KRBTEST.COM; defaulting to no policy
I20260806 02:47:45.416572 13303 ts_manager.cc:194] Registered new tserver with Master: 89701522673d4020993463f9228b73ee (127.1.210.66:45753)
I20260806 02:47:45.417757 13303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.66@KRBTEST.COM'} at 127.1.210.66:36781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab.
Entry for principal kudu/127.1.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab.
I20260806 02:47:45.459156 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.210.84:53127/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.67:0
--local_ip_for_outbound_sockets=127.1.210.67
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:34347
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:38245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:47:46.128890 13665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:47:46.129294 13665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:47:46.129591 13665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:47:46.140591 13665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:47:46.140801 13665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:47:46.141016 13665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.67
I20260806 02:47:46.155809 13665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:38245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.210.84:53127/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:34347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.67
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:47:46.157434 13665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:47:46.159205 13665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:47:46.181180 13670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:47:46.181543 13671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:47:46.181613 13673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:47:46.184204 13665 server_base.cc:1034] running on GCE node
I20260806 02:47:46.184971 13665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:47:46.186363 13665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:47:46.187747 13665 hybrid_clock.cc:648] HybridClock initialized: now 1785984466187605 us; error 64 us; skew 500 ppm
I20260806 02:47:46.188280 13665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:47:46 dist-test-slave-8tdl krb5kdc[12827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984466, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:47:46.194589 13665 init.cc:377] Logged in from keytab as kudu/127.1.210.67@KRBTEST.COM (short username kudu)
I20260806 02:47:46.197651 13665 webserver.cc:460] Webserver started at http://127.1.210.67:38777/ using document root <none> and password file <none>
I20260806 02:47:46.198663 13665 fs_manager.cc:359] Metadata directory not provided
I20260806 02:47:46.198938 13665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:47:46.199369 13665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:47:46 dist-test-slave-8tdl krb5kdc[12827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984466, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for HTTP/127.1.210.84@KRBTEST.COM
I20260806 02:47:46.218597 13665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-2/data/instance:
uuid: "81f5ca2c7e5e4531b9211808bc3b03a2"
format_stamp: "Formatted at 2026-08-06 02:47:46 on dist-test-slave-8tdl"
server_key: "7bbbef26b82e5cab1dc5e592091b406e"
server_key_iv: "5c1c85c19ccee413fde7b1426857545a"
server_key_version: "kuduclusterkey@0"
I20260806 02:47:46.220101 13665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance:
uuid: "81f5ca2c7e5e4531b9211808bc3b03a2"
format_stamp: "Formatted at 2026-08-06 02:47:46 on dist-test-slave-8tdl"
server_key: "7bbbef26b82e5cab1dc5e592091b406e"
server_key_iv: "5c1c85c19ccee413fde7b1426857545a"
server_key_version: "kuduclusterkey@0"
I20260806 02:47:46.229066 13665 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.002s sys 0.006s
I20260806 02:47:46.247083 13680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:47:46.249330 13665 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260806 02:47:46.249585 13665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-2/wal
uuid: "81f5ca2c7e5e4531b9211808bc3b03a2"
format_stamp: "Formatted at 2026-08-06 02:47:46 on dist-test-slave-8tdl"
server_key: "7bbbef26b82e5cab1dc5e592091b406e"
server_key_iv: "5c1c85c19ccee413fde7b1426857545a"
server_key_version: "kuduclusterkey@0"
I20260806 02:47:46.249905 13665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:47:46.279027 13665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:47:46.284600 13665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:47:46.285212 13665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:47:46.287238 13665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:47:46.289660 13665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:47:46.289777 13665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:47:46.289899 13665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:47:46.289989 13665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:47:46.378197 13665 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.67:45197
I20260806 02:47:46.378301 13792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.67:45197 every 8 connection(s)
I20260806 02:47:46.380256 13665 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
I20260806 02:47:46.390571 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 13665
I20260806 02:47:46.390810 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance
I20260806 02:47:46.391235 1865 external_mini_cluster.cc:1428] Setting key 5191c50c9204768137efcfb823316a44
Aug 06 02:47:46 dist-test-slave-8tdl krb5kdc[12827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984466, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:47:46.409909 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:46.384143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:57751 (local address 127.1.210.126:34347)
0806 02:47:46.384956 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0806 02:47:46.384963 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:47:46.386291 (+ 1328us) server_negotiation.cc:407] Connection header received
0806 02:47:46.389071 (+ 2780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:47:46.389076 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:47:46.389150 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:47:46.389257 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:47:46.391725 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:46.393106 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:46.395852 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:46.396506 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:46.396641 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:46.401284 (+ 4643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:47:46.401289 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:47:46.401293 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:47:46.401343 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:47:46.404823 (+ 3480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:47:46.406211 (+ 1388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:47:46.406249 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:47:46.406252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:47:46.406449 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:47:46.407377 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:47:46.407381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:47:46.407383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:47:46.407962 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:47:46.408215 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0806 02:47:46.409129 (+ 914us) server_negotiation.cc:299] Negotiation successful
0806 02:47:46.409385 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":483,"threads_started":1}
I20260806 02:47:46.411226 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:46.384945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:34347 (local address 127.1.210.67:57751)
0806 02:47:46.386096 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0806 02:47:46.386152 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 02:47:46.388688 (+ 2536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:47:46.389637 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:47:46.389646 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:47:46.390375 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:47:46.391485 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:47:46.391507 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:47:46.393293 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:47:46.393297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:47:46.395694 (+ 2397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:47:46.395708 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:47:46.396867 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:47:46.396874 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:47:46.397180 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:46.398098 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:47:46.398124 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:47:46.401003 (+ 2879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:47:46.405031 (+ 4028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:47:46.405040 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:47:46.405055 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:47:46.406023 (+ 968us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:47:46.406582 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:47:46.406588 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:47:46.406591 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:47:46.407225 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:47:46.408175 (+ 950us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:47:46.408186 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:47:46.408690 (+ 504us) client_negotiation.cc:769] Sending connection context
0806 02:47:46.408966 (+ 276us) client_negotiation.cc:240] Negotiation successful
0806 02:47:46.409459 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"thread_start_us":569,"threads_started":1}
I20260806 02:47:46.413208 13793 heartbeater.cc:347] Connected to a master server at 127.1.210.126:34347
I20260806 02:47:46.413599 13793 heartbeater.cc:464] Registering TS with master...
I20260806 02:47:46.414557 13793 heartbeater.cc:511] Master 127.1.210.126:34347 requested a full tablet report, sending...
I20260806 02:47:46.416520 13303 ts_manager.cc:194] Registered new tserver with Master: 81f5ca2c7e5e4531b9211808bc3b03a2 (127.1.210.67:45197)
I20260806 02:47:46.417734 13303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.67@KRBTEST.COM'} at 127.1.210.67:57751
I20260806 02:47:46.421129 13659 heartbeater.cc:503] Master 127.1.210.126:34347 was elected leader, sending a full tablet report...
I20260806 02:47:46.421662 1865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:47:46.447290 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:46.428997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37892 (local address 127.1.210.126:34347)
0806 02:47:46.429330 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0806 02:47:46.429340 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:47:46.429958 (+ 618us) server_negotiation.cc:407] Connection header received
0806 02:47:46.430208 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:47:46.430219 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:47:46.430319 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:47:46.430443 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:47:46.432596 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:46.433637 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:46.435672 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:46.436341 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:46.436507 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:46.440004 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:47:46.440010 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:47:46.440014 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:47:46.440050 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:47:46.443817 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:47:46.444773 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:47:46.444778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:47:46.444780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:47:46.444969 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:47:46.445485 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:47:46.445493 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:47:46.445497 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:47:46.445791 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:47:46.446165 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0806 02:47:46.446513 (+ 348us) server_negotiation.cc:299] Negotiation successful
0806 02:47:46.446892 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260806 02:47:46.452523 13303 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:47:46.462177 13303 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:47:46.516556 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:46.482829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:46093 (local address 127.0.0.1:42340)
0806 02:47:46.502511 (+ 19682us) negotiation.cc:107] Waiting for socket to connect
0806 02:47:46.502530 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:47:46.502959 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:47:46.504209 (+ 1250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:47:46.504216 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:47:46.504250 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:47:46.505349 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:47:46.505367 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:47:46.507889 (+ 2522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:47:46.507897 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:47:46.512262 (+ 4365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:47:46.512276 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:47:46.514361 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:47:46.514370 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:47:46.514648 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:46.514742 (+ 94us) client_negotiation.cc:769] Sending connection context
0806 02:47:46.514953 (+ 211us) client_negotiation.cc:240] Negotiation successful
0806 02:47:46.515222 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19477,"mutex_wait_us":198,"thread_start_us":17795,"threads_started":1}
I20260806 02:47:46.517758 13810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:46.483279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42340 (local address 127.1.210.65:46093)
0806 02:47:46.492853 (+ 9574us) server_negotiation.cc:206] Beginning negotiation
0806 02:47:46.492869 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:47:46.502731 (+ 9862us) server_negotiation.cc:407] Connection header received
0806 02:47:46.503202 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:47:46.503211 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:47:46.503514 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:47:46.503744 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:47:46.505628 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:46.507742 (+ 2114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:46.512445 (+ 4703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:46.513995 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:46.514133 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:46.514556 (+ 423us) server_negotiation.cc:1035] Waiting for connection context
0806 02:47:46.515041 (+ 485us) server_negotiation.cc:299] Negotiation successful
0806 02:47:46.517120 (+ 2079us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9287,"thread_start_us":9003,"threads_started":1}
I20260806 02:47:46.518735 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:46.482786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54866 (local address 127.1.210.67:45197)
0806 02:47:46.499743 (+ 16957us) server_negotiation.cc:206] Beginning negotiation
0806 02:47:46.499757 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:47:46.502124 (+ 2367us) server_negotiation.cc:407] Connection header received
0806 02:47:46.502534 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:47:46.502545 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:47:46.502962 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:47:46.503203 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:47:46.504712 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:46.507419 (+ 2707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:46.513189 (+ 5770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:46.515315 (+ 2126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:46.515501 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:46.515866 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0806 02:47:46.517858 (+ 1992us) server_negotiation.cc:299] Negotiation successful
0806 02:47:46.518181 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16549,"thread_start_us":16179,"threads_started":1}
I20260806 02:47:46.518764 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:46.492499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60782 (local address 127.1.210.66:45753)
0806 02:47:46.504863 (+ 12364us) server_negotiation.cc:206] Beginning negotiation
0806 02:47:46.504872 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:47:46.504894 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:47:46.504976 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:47:46.504982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:47:46.505186 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:47:46.505425 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:47:46.506728 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:46.508862 (+ 2134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:46.513620 (+ 4758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:46.515923 (+ 2303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:46.516301 (+ 378us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:46.516643 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0806 02:47:46.517931 (+ 1288us) server_negotiation.cc:299] Negotiation successful
0806 02:47:46.518198 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12209,"thread_start_us":10871,"threads_started":1}
I20260806 02:47:46.519518 13809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:46.482519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:45197 (local address 127.0.0.1:54866)
0806 02:47:46.501589 (+ 19070us) negotiation.cc:107] Waiting for socket to connect
0806 02:47:46.501674 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0806 02:47:46.502030 (+ 356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:47:46.503271 (+ 1241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:47:46.503280 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:47:46.503314 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:47:46.504385 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:47:46.504398 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:47:46.507615 (+ 3217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:47:46.507621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:47:46.513004 (+ 5383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:47:46.513020 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:47:46.516353 (+ 3333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:47:46.516362 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:47:46.516718 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:46.516762 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 02:47:46.516951 (+ 189us) client_negotiation.cc:240] Negotiation successful
0806 02:47:46.517136 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18606,"mutex_wait_us":112,"thread_start_us":19631,"threads_started":1}
I20260806 02:47:46.519562 13811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:46.491629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:45753 (local address 127.0.0.1:60782)
0806 02:47:46.504138 (+ 12509us) negotiation.cc:107] Waiting for socket to connect
0806 02:47:46.504172 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 02:47:46.504370 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:47:46.505407 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:47:46.505416 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:47:46.505456 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:47:46.506425 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:47:46.506437 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:47:46.509253 (+ 2816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:47:46.509260 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:47:46.513450 (+ 4190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:47:46.513468 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:47:46.516984 (+ 3516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:47:46.516994 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:47:46.517281 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:46.517342 (+ 61us) client_negotiation.cc:769] Sending connection context
0806 02:47:46.517440 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 02:47:46.517596 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12299,"mutex_wait_us":1,"thread_start_us":10977,"threads_started":1}
I20260806 02:47:46.522572 13451 tablet_service.cc:1467] Processing CreateTablet for tablet d236fd4bd9274b4398255f96d710810a (DEFAULT_TABLE table=test-table [id=b19989ec53784496a7f1c875f388134b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:47:46.523949 13594 tablet_service.cc:1467] Processing CreateTablet for tablet d236fd4bd9274b4398255f96d710810a (DEFAULT_TABLE table=test-table [id=b19989ec53784496a7f1c875f388134b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:47:46.525746 13451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d236fd4bd9274b4398255f96d710810a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:47:46.525789 13594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d236fd4bd9274b4398255f96d710810a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:47:46.527663 13728 tablet_service.cc:1467] Processing CreateTablet for tablet d236fd4bd9274b4398255f96d710810a (DEFAULT_TABLE table=test-table [id=b19989ec53784496a7f1c875f388134b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:47:46.529753 13728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d236fd4bd9274b4398255f96d710810a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:47:46.543036 13813 tablet_bootstrap.cc:492] T d236fd4bd9274b4398255f96d710810a P 89701522673d4020993463f9228b73ee: Bootstrap starting.
I20260806 02:47:46.547734 13814 tablet_bootstrap.cc:492] T d236fd4bd9274b4398255f96d710810a P 8129982a4d914f71b531306b72dacab0: Bootstrap starting.
I20260806 02:47:46.548928 13813 tablet_bootstrap.cc:654] T d236fd4bd9274b4398255f96d710810a P 89701522673d4020993463f9228b73ee: Neither blocks nor log segments found. Creating new log.
I20260806 02:47:46.551189 13813 log.cc:824] T d236fd4bd9274b4398255f96d710810a P 89701522673d4020993463f9228b73ee: Log is configured to *not* fsync() on all Append() calls
I20260806 02:47:46.551307 13815 tablet_bootstrap.cc:492] T d236fd4bd9274b4398255f96d710810a P 81f5ca2c7e5e4531b9211808bc3b03a2: Bootstrap starting.
I20260806 02:47:46.553339 13814 tablet_bootstrap.cc:654] T d236fd4bd9274b4398255f96d710810a P 8129982a4d914f71b531306b72dacab0: Neither blocks nor log segments found. Creating new log.
I20260806 02:47:46.555697 13814 log.cc:824] T d236fd4bd9274b4398255f96d710810a P 8129982a4d914f71b531306b72dacab0: Log is configured to *not* fsync() on all Append() calls
I20260806 02:47:46.556871 13815 tablet_bootstrap.cc:654] T d236fd4bd9274b4398255f96d710810a P 81f5ca2c7e5e4531b9211808bc3b03a2: Neither blocks nor log segments found. Creating new log.
I20260806 02:47:46.557228 13813 tablet_bootstrap.cc:492] T d236fd4bd9274b4398255f96d710810a P 89701522673d4020993463f9228b73ee: No bootstrap required, opened a new log
I20260806 02:47:46.557693 13813 ts_tablet_manager.cc:1397] T d236fd4bd9274b4398255f96d710810a P 89701522673d4020993463f9228b73ee: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.007s
I20260806 02:47:46.559331 13815 log.cc:824] T d236fd4bd9274b4398255f96d710810a P 81f5ca2c7e5e4531b9211808bc3b03a2: Log is configured to *not* fsync() on all Append() calls
I20260806 02:47:46.561470 13814 tablet_bootstrap.cc:492] T d236fd4bd9274b4398255f96d710810a P 8129982a4d914f71b531306b72dacab0: No bootstrap required, opened a new log
I20260806 02:47:46.561869 13814 ts_tablet_manager.cc:1397] T d236fd4bd9274b4398255f96d710810a P 8129982a4d914f71b531306b72dacab0: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260806 02:47:46.563599 13815 tablet_bootstrap.cc:492] T d236fd4bd9274b4398255f96d710810a P 81f5ca2c7e5e4531b9211808bc3b03a2: No bootstrap required, opened a new log
I20260806 02:47:46.563923 13815 ts_tablet_manager.cc:1397] T d236fd4bd9274b4398255f96d710810a P 81f5ca2c7e5e4531b9211808bc3b03a2: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260806 02:47:46.572741 13813 raft_consensus.cc:348] T d236fd4bd9274b4398255f96d710810a P 89701522673d4020993463f9228b73ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8129982a4d914f71b531306b72dacab0" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46093 } } peers { permanent_uuid: "81f5ca2c7e5e4531b9211808bc3b03a2" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45197 } } peers { permanent_uuid: "89701522673d4020993463f9228b73ee" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45753 } }
I20260806 02:47:46.573467 13813 raft_consensus.cc:374] T d236fd4bd9274b4398255f96d710810a P 89701522673d4020993463f9228b73ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:47:46.573638 13813 raft_consensus.cc:729] T d236fd4bd9274b4398255f96d710810a P 89701522673d4020993463f9228b73ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89701522673d4020993463f9228b73ee, State: Initialized, Role: FOLLOWER
I20260806 02:47:46.574178 13815 raft_consensus.cc:348] T d236fd4bd9274b4398255f96d710810a P 81f5ca2c7e5e4531b9211808bc3b03a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8129982a4d914f71b531306b72dacab0" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46093 } } peers { permanent_uuid: "81f5ca2c7e5e4531b9211808bc3b03a2" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45197 } } peers { permanent_uuid: "89701522673d4020993463f9228b73ee" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45753 } }
I20260806 02:47:46.574790 13815 raft_consensus.cc:374] T d236fd4bd9274b4398255f96d710810a P 81f5ca2c7e5e4531b9211808bc3b03a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:47:46.574580 13813 consensus_queue.cc:260] T d236fd4bd9274b4398255f96d710810a P 89701522673d4020993463f9228b73ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8129982a4d914f71b531306b72dacab0" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46093 } } peers { permanent_uuid: "81f5ca2c7e5e4531b9211808bc3b03a2" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45197 } } peers { permanent_uuid: "89701522673d4020993463f9228b73ee" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45753 } }
I20260806 02:47:46.574962 13815 raft_consensus.cc:729] T d236fd4bd9274b4398255f96d710810a P 81f5ca2c7e5e4531b9211808bc3b03a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81f5ca2c7e5e4531b9211808bc3b03a2, State: Initialized, Role: FOLLOWER
I20260806 02:47:46.576046 13815 consensus_queue.cc:260] T d236fd4bd9274b4398255f96d710810a P 81f5ca2c7e5e4531b9211808bc3b03a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8129982a4d914f71b531306b72dacab0" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46093 } } peers { permanent_uuid: "81f5ca2c7e5e4531b9211808bc3b03a2" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45197 } } peers { permanent_uuid: "89701522673d4020993463f9228b73ee" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45753 } }
I20260806 02:47:46.577368 13813 ts_tablet_manager.cc:1428] T d236fd4bd9274b4398255f96d710810a P 89701522673d4020993463f9228b73ee: Time spent starting tablet: real 0.019s user 0.011s sys 0.007s
I20260806 02:47:46.581573 13814 raft_consensus.cc:348] T d236fd4bd9274b4398255f96d710810a P 8129982a4d914f71b531306b72dacab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8129982a4d914f71b531306b72dacab0" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46093 } } peers { permanent_uuid: "81f5ca2c7e5e4531b9211808bc3b03a2" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45197 } } peers { permanent_uuid: "89701522673d4020993463f9228b73ee" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45753 } }
I20260806 02:47:46.582227 13814 raft_consensus.cc:374] T d236fd4bd9274b4398255f96d710810a P 8129982a4d914f71b531306b72dacab0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:47:46.582473 13814 raft_consensus.cc:729] T d236fd4bd9274b4398255f96d710810a P 8129982a4d914f71b531306b72dacab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8129982a4d914f71b531306b72dacab0, State: Initialized, Role: FOLLOWER
I20260806 02:47:46.583392 13814 consensus_queue.cc:260] T d236fd4bd9274b4398255f96d710810a P 8129982a4d914f71b531306b72dacab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8129982a4d914f71b531306b72dacab0" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46093 } } peers { permanent_uuid: "81f5ca2c7e5e4531b9211808bc3b03a2" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45197 } } peers { permanent_uuid: "89701522673d4020993463f9228b73ee" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45753 } }
I20260806 02:47:46.585383 13815 ts_tablet_manager.cc:1428] T d236fd4bd9274b4398255f96d710810a P 81f5ca2c7e5e4531b9211808bc3b03a2: Time spent starting tablet: real 0.021s user 0.013s sys 0.007s
I20260806 02:47:46.585815 13814 ts_tablet_manager.cc:1428] T d236fd4bd9274b4398255f96d710810a P 8129982a4d914f71b531306b72dacab0: Time spent starting tablet: real 0.024s user 0.007s sys 0.013s
I20260806 02:47:46.585960 13793 heartbeater.cc:503] Master 127.1.210.126:34347 was elected leader, sending a full tablet report...
W20260806 02:47:46.616532 13517 tablet.cc:2369] T d236fd4bd9274b4398255f96d710810a P 8129982a4d914f71b531306b72dacab0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:47:46.627288 13660 tablet.cc:2369] T d236fd4bd9274b4398255f96d710810a P 89701522673d4020993463f9228b73ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:47:46.633252 13794 tablet.cc:2369] T d236fd4bd9274b4398255f96d710810a P 81f5ca2c7e5e4531b9211808bc3b03a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:47:46.782972 13820 raft_consensus.cc:482] T d236fd4bd9274b4398255f96d710810a P 81f5ca2c7e5e4531b9211808bc3b03a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:47:46.783485 13820 raft_consensus.cc:504] T d236fd4bd9274b4398255f96d710810a P 81f5ca2c7e5e4531b9211808bc3b03a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8129982a4d914f71b531306b72dacab0" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46093 } } peers { permanent_uuid: "81f5ca2c7e5e4531b9211808bc3b03a2" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45197 } } peers { permanent_uuid: "89701522673d4020993463f9228b73ee" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45753 } }
I20260806 02:47:46.785885 13820 leader_election.cc:288] T d236fd4bd9274b4398255f96d710810a P 81f5ca2c7e5e4531b9211808bc3b03a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8129982a4d914f71b531306b72dacab0 (127.1.210.65:46093), 89701522673d4020993463f9228b73ee (127.1.210.66:45753)
I20260806 02:47:46.795073 13810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:46.786656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:41991 (local address 127.1.210.65:46093)
0806 02:47:46.787010 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0806 02:47:46.787017 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:47:46.787275 (+ 258us) server_negotiation.cc:407] Connection header received
0806 02:47:46.787535 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:47:46.787541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:47:46.787637 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:47:46.787753 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:47:46.788304 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:46.789321 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:46.791984 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:46.793918 (+ 1934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:46.794171 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:46.794298 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 02:47:46.794586 (+ 288us) server_negotiation.cc:299] Negotiation successful
0806 02:47:46.794761 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260806 02:47:46.795130 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:46.786493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:46093 (local address 127.1.210.67:41991)
0806 02:47:46.787106 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0806 02:47:46.787129 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:47:46.787305 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:47:46.787804 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:47:46.787810 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:47:46.787846 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:47:46.788136 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:47:46.788146 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:47:46.789551 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:47:46.789556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:47:46.791768 (+ 2212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:47:46.791783 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:47:46.794177 (+ 2394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:47:46.794183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:47:46.794357 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:46.794386 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 02:47:46.794547 (+ 161us) client_negotiation.cc:240] Negotiation successful
0806 02:47:46.794733 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454}
I20260806 02:47:46.797132 13471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d236fd4bd9274b4398255f96d710810a" candidate_uuid: "81f5ca2c7e5e4531b9211808bc3b03a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8129982a4d914f71b531306b72dacab0" is_pre_election: true
I20260806 02:47:46.797850 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:46.786741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:43703 (local address 127.1.210.66:45753)
0806 02:47:46.787056 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0806 02:47:46.787062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:47:46.788053 (+ 991us) server_negotiation.cc:407] Connection header received
0806 02:47:46.788269 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:47:46.788281 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:47:46.788343 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:47:46.788477 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:47:46.789504 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:46.790869 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:46.794576 (+ 3707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:46.796375 (+ 1799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:46.796578 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:46.796636 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 02:47:46.797382 (+ 746us) server_negotiation.cc:299] Negotiation successful
0806 02:47:46.797531 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 02:47:46.797894 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:46.786775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:45753 (local address 127.1.210.67:43703)
0806 02:47:46.787864 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0806 02:47:46.787882 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:47:46.788123 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:47:46.788493 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:47:46.788497 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:47:46.788522 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:47:46.789310 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:47:46.789322 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:47:46.791071 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:47:46.791078 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:47:46.794324 (+ 3246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:47:46.794345 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:47:46.796835 (+ 2490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:47:46.796843 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:47:46.797128 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:46.797172 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 02:47:46.797293 (+ 121us) client_negotiation.cc:240] Negotiation successful
0806 02:47:46.797487 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":936,"thread_start_us":769,"threads_started":1}
I20260806 02:47:46.798019 13471 raft_consensus.cc:2454] T d236fd4bd9274b4398255f96d710810a P 8129982a4d914f71b531306b72dacab0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81f5ca2c7e5e4531b9211808bc3b03a2 in term 0.
I20260806 02:47:46.799213 13683 leader_election.cc:302] T d236fd4bd9274b4398255f96d710810a P 81f5ca2c7e5e4531b9211808bc3b03a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8129982a4d914f71b531306b72dacab0, 81f5ca2c7e5e4531b9211808bc3b03a2; no voters:
I20260806 02:47:46.799837 13820 raft_consensus.cc:2781] T d236fd4bd9274b4398255f96d710810a P 81f5ca2c7e5e4531b9211808bc3b03a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:47:46.800004 13820 raft_consensus.cc:482] T d236fd4bd9274b4398255f96d710810a P 81f5ca2c7e5e4531b9211808bc3b03a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:47:46.799795 13614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d236fd4bd9274b4398255f96d710810a" candidate_uuid: "81f5ca2c7e5e4531b9211808bc3b03a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89701522673d4020993463f9228b73ee" is_pre_election: true
I20260806 02:47:46.800163 13820 raft_consensus.cc:3037] T d236fd4bd9274b4398255f96d710810a P 81f5ca2c7e5e4531b9211808bc3b03a2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:47:46.800611 13614 raft_consensus.cc:2454] T d236fd4bd9274b4398255f96d710810a P 89701522673d4020993463f9228b73ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81f5ca2c7e5e4531b9211808bc3b03a2 in term 0.
I20260806 02:47:46.804311 13820 raft_consensus.cc:504] T d236fd4bd9274b4398255f96d710810a P 81f5ca2c7e5e4531b9211808bc3b03a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8129982a4d914f71b531306b72dacab0" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46093 } } peers { permanent_uuid: "81f5ca2c7e5e4531b9211808bc3b03a2" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45197 } } peers { permanent_uuid: "89701522673d4020993463f9228b73ee" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45753 } }
I20260806 02:47:46.805298 13820 leader_election.cc:288] T d236fd4bd9274b4398255f96d710810a P 81f5ca2c7e5e4531b9211808bc3b03a2 [CANDIDATE]: Term 1 election: Requested vote from peers 8129982a4d914f71b531306b72dacab0 (127.1.210.65:46093), 89701522673d4020993463f9228b73ee (127.1.210.66:45753)
I20260806 02:47:46.805903 13471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d236fd4bd9274b4398255f96d710810a" candidate_uuid: "81f5ca2c7e5e4531b9211808bc3b03a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8129982a4d914f71b531306b72dacab0"
I20260806 02:47:46.805948 13614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d236fd4bd9274b4398255f96d710810a" candidate_uuid: "81f5ca2c7e5e4531b9211808bc3b03a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89701522673d4020993463f9228b73ee"
I20260806 02:47:46.806187 13471 raft_consensus.cc:3037] T d236fd4bd9274b4398255f96d710810a P 8129982a4d914f71b531306b72dacab0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:47:46.806293 13614 raft_consensus.cc:3037] T d236fd4bd9274b4398255f96d710810a P 89701522673d4020993463f9228b73ee [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:47:46.809274 13471 raft_consensus.cc:2454] T d236fd4bd9274b4398255f96d710810a P 8129982a4d914f71b531306b72dacab0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81f5ca2c7e5e4531b9211808bc3b03a2 in term 1.
I20260806 02:47:46.809561 13614 raft_consensus.cc:2454] T d236fd4bd9274b4398255f96d710810a P 89701522673d4020993463f9228b73ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81f5ca2c7e5e4531b9211808bc3b03a2 in term 1.
I20260806 02:47:46.809914 13683 leader_election.cc:302] T d236fd4bd9274b4398255f96d710810a P 81f5ca2c7e5e4531b9211808bc3b03a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8129982a4d914f71b531306b72dacab0, 81f5ca2c7e5e4531b9211808bc3b03a2; no voters:
I20260806 02:47:46.810328 13820 raft_consensus.cc:2781] T d236fd4bd9274b4398255f96d710810a P 81f5ca2c7e5e4531b9211808bc3b03a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:47:46.811270 13820 raft_consensus.cc:686] T d236fd4bd9274b4398255f96d710810a P 81f5ca2c7e5e4531b9211808bc3b03a2 [term 1 LEADER]: Becoming Leader. State: Replica: 81f5ca2c7e5e4531b9211808bc3b03a2, State: Running, Role: LEADER
I20260806 02:47:46.811995 13820 consensus_queue.cc:237] T d236fd4bd9274b4398255f96d710810a P 81f5ca2c7e5e4531b9211808bc3b03a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8129982a4d914f71b531306b72dacab0" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46093 } } peers { permanent_uuid: "81f5ca2c7e5e4531b9211808bc3b03a2" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45197 } } peers { permanent_uuid: "89701522673d4020993463f9228b73ee" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45753 } }
I20260806 02:47:46.817649 13302 catalog_manager.cc:5495] T d236fd4bd9274b4398255f96d710810a P 81f5ca2c7e5e4531b9211808bc3b03a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 81f5ca2c7e5e4531b9211808bc3b03a2 (127.1.210.67). New cstate: current_term: 1 leader_uuid: "81f5ca2c7e5e4531b9211808bc3b03a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8129982a4d914f71b531306b72dacab0" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81f5ca2c7e5e4531b9211808bc3b03a2" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89701522673d4020993463f9228b73ee" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45753 } health_report { overall_health: UNKNOWN } } }
I20260806 02:47:46.878556 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:46.871731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54882 (local address 127.1.210.67:45197)
0806 02:47:46.872242 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0806 02:47:46.872247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:47:46.872269 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:47:46.872364 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:47:46.872368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:47:46.872457 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:47:46.872592 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:47:46.873096 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:46.874016 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:46.876039 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:46.876634 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:46.876743 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:46.877372 (+ 629us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:47:46.877906 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:47:46.878111 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0806 02:47:46.878158 (+ 47us) server_negotiation.cc:299] Negotiation successful
0806 02:47:46.878234 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333}
I20260806 02:47:46.889359 13614 raft_consensus.cc:1261] T d236fd4bd9274b4398255f96d710810a P 89701522673d4020993463f9228b73ee [term 1 FOLLOWER]: Refusing update from remote peer 81f5ca2c7e5e4531b9211808bc3b03a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:47:46.889639 13471 raft_consensus.cc:1261] T d236fd4bd9274b4398255f96d710810a P 8129982a4d914f71b531306b72dacab0 [term 1 FOLLOWER]: Refusing update from remote peer 81f5ca2c7e5e4531b9211808bc3b03a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:47:46.890823 13820 consensus_queue.cc:1035] T d236fd4bd9274b4398255f96d710810a P 81f5ca2c7e5e4531b9211808bc3b03a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8129982a4d914f71b531306b72dacab0" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:47:46.891304 13823 consensus_queue.cc:1035] T d236fd4bd9274b4398255f96d710810a P 81f5ca2c7e5e4531b9211808bc3b03a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89701522673d4020993463f9228b73ee" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:47:46.928073 13302 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:47:46.928555 13302 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:47:46.933601 13302 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f387ce1ddc944ec973e0eac647a9986: Sending DeleteTablet for 3 replicas of tablet d236fd4bd9274b4398255f96d710810a
I20260806 02:47:46.935281 13451 tablet_service.cc:1514] Processing DeleteTablet for tablet d236fd4bd9274b4398255f96d710810a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:47:46 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:42340
I20260806 02:47:46.935287 13728 tablet_service.cc:1514] Processing DeleteTablet for tablet d236fd4bd9274b4398255f96d710810a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:47:46 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:54866
I20260806 02:47:46.935349 13594 tablet_service.cc:1514] Processing DeleteTablet for tablet d236fd4bd9274b4398255f96d710810a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:47:46 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:60782
I20260806 02:47:46.940496 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 13383
I20260806 02:47:46.946666 13835 tablet_replica.cc:330] T d236fd4bd9274b4398255f96d710810a P 8129982a4d914f71b531306b72dacab0: stopping tablet replica
I20260806 02:47:46.947355 13835 raft_consensus.cc:2229] T d236fd4bd9274b4398255f96d710810a P 8129982a4d914f71b531306b72dacab0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:47:46.948302 13835 raft_consensus.cc:2258] T d236fd4bd9274b4398255f96d710810a P 8129982a4d914f71b531306b72dacab0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:47:46.951298 13835 ts_tablet_manager.cc:1905] T d236fd4bd9274b4398255f96d710810a P 8129982a4d914f71b531306b72dacab0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:47:46.951504 13836 tablet_replica.cc:330] T d236fd4bd9274b4398255f96d710810a P 89701522673d4020993463f9228b73ee: stopping tablet replica
I20260806 02:47:46.952142 13836 raft_consensus.cc:2229] T d236fd4bd9274b4398255f96d710810a P 89701522673d4020993463f9228b73ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:47:46.952986 13836 raft_consensus.cc:2258] T d236fd4bd9274b4398255f96d710810a P 89701522673d4020993463f9228b73ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:47:46.955395 13836 ts_tablet_manager.cc:1905] T d236fd4bd9274b4398255f96d710810a P 89701522673d4020993463f9228b73ee: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 06 02:47:46 dist-test-slave-8tdl krb5kdc[12827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.65@KRBTEST.COM
I20260806 02:47:46.965857 13836 ts_tablet_manager.cc:1918] T d236fd4bd9274b4398255f96d710810a P 89701522673d4020993463f9228b73ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:47:46.966202 13836 log.cc:1196] T d236fd4bd9274b4398255f96d710810a P 89701522673d4020993463f9228b73ee: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-1/wal/wals/d236fd4bd9274b4398255f96d710810a
I20260806 02:47:46.967146 13836 ts_tablet_manager.cc:1939] T d236fd4bd9274b4398255f96d710810a P 89701522673d4020993463f9228b73ee: Deleting consensus metadata
I20260806 02:47:46.970249 13288 catalog_manager.cc:4841] TS 89701522673d4020993463f9228b73ee (127.1.210.66:45753): tablet d236fd4bd9274b4398255f96d710810a (table test-table [id=b19989ec53784496a7f1c875f388134b]) successfully deleted
I20260806 02:47:46.971680 13837 tablet_replica.cc:330] T d236fd4bd9274b4398255f96d710810a P 81f5ca2c7e5e4531b9211808bc3b03a2: stopping tablet replica
I20260806 02:47:46.972903 13837 raft_consensus.cc:2229] T d236fd4bd9274b4398255f96d710810a P 81f5ca2c7e5e4531b9211808bc3b03a2 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:47:46.973820 13837 raft_consensus.cc:2258] T d236fd4bd9274b4398255f96d710810a P 81f5ca2c7e5e4531b9211808bc3b03a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:47:46.976366 13837 ts_tablet_manager.cc:1905] T d236fd4bd9274b4398255f96d710810a P 81f5ca2c7e5e4531b9211808bc3b03a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:47:46.978626 13835 ts_tablet_manager.cc:1918] T d236fd4bd9274b4398255f96d710810a P 8129982a4d914f71b531306b72dacab0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:47:46.978899 13835 log.cc:1196] T d236fd4bd9274b4398255f96d710810a P 8129982a4d914f71b531306b72dacab0: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-0/wal/wals/d236fd4bd9274b4398255f96d710810a
I20260806 02:47:46.979700 13810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:46.942150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42344 (local address 127.1.210.65:46093)
0806 02:47:46.942363 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 02:47:46.942370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:47:46.950260 (+ 7890us) server_negotiation.cc:407] Connection header received
0806 02:47:46.950483 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:47:46.950490 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:47:46.950623 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:47:46.950816 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:47:46.953523 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:46.954914 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:46.957608 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:46.958472 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:46.958587 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:46.966208 (+ 7621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:47:46.966237 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:47:46.966258 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:47:46.966314 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:47:46.973366 (+ 7052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:47:46.974771 (+ 1405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:47:46.974781 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:47:46.974785 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:47:46.975028 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:47:46.975637 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:47:46.975643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:47:46.975646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:47:46.976345 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:47:46.976809 (+ 464us) server_negotiation.cc:1035] Waiting for connection context
0806 02:47:46.978195 (+ 1386us) server_negotiation.cc:299] Negotiation successful
0806 02:47:46.978562 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 02:47:46.980023 13835 ts_tablet_manager.cc:1939] T d236fd4bd9274b4398255f96d710810a P 8129982a4d914f71b531306b72dacab0: Deleting consensus metadata
I20260806 02:47:46.981158 13511 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:47:46.986860 13837 ts_tablet_manager.cc:1918] T d236fd4bd9274b4398255f96d710810a P 81f5ca2c7e5e4531b9211808bc3b03a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:47:46.987176 13837 log.cc:1196] T d236fd4bd9274b4398255f96d710810a P 81f5ca2c7e5e4531b9211808bc3b03a2: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984304136196-1865-0/minicluster-data/ts-2/wal/wals/d236fd4bd9274b4398255f96d710810a
I20260806 02:47:46.988473 13837 ts_tablet_manager.cc:1939] T d236fd4bd9274b4398255f96d710810a P 81f5ca2c7e5e4531b9211808bc3b03a2: Deleting consensus metadata
I20260806 02:47:46.991124 13290 catalog_manager.cc:4841] TS 81f5ca2c7e5e4531b9211808bc3b03a2 (127.1.210.67:45197): tablet d236fd4bd9274b4398255f96d710810a (table test-table [id=b19989ec53784496a7f1c875f388134b]) successfully deleted
I20260806 02:47:47.082420 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 13383
W20260806 02:47:47.103507 13289 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.210.65:46093 (error 108)
I20260806 02:47:47.103760 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 13522
I20260806 02:47:47.107815 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:47.106138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:46093 (local address 127.0.0.1:42358)
0806 02:47:47.107302 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0806 02:47:47.107520 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":987,"thread_start_us":871,"threads_started":1}
W20260806 02:47:47.108115 13289 catalog_manager.cc:4584] TS 8129982a4d914f71b531306b72dacab0 (127.1.210.65:46093): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d236fd4bd9274b4398255f96d710810a: Network error: Client connection negotiation failed: client connection to 127.1.210.65:46093: connect: Connection refused (error 111)
Aug 06 02:47:47 dist-test-slave-8tdl krb5kdc[12827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.66@KRBTEST.COM
I20260806 02:47:47.123103 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:47.104789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60790 (local address 127.1.210.66:45753)
0806 02:47:47.105048 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0806 02:47:47.105058 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:47:47.105086 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:47:47.105184 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:47:47.105193 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:47:47.105302 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:47:47.105431 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:47:47.107540 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:47.108433 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:47.109875 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:47.110306 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:47.110471 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:47.115655 (+ 5184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:47:47.115672 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:47:47.115688 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:47:47.115740 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:47:47.119109 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:47:47.120166 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:47:47.120178 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:47:47.120183 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:47:47.120390 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:47:47.121044 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:47:47.121051 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:47:47.121057 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:47:47.121609 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:47:47.121986 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0806 02:47:47.122335 (+ 349us) server_negotiation.cc:299] Negotiation successful
0806 02:47:47.122613 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 02:47:47.123947 13654 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:47:47.162139 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:47.161549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:46093 (local address 127.0.0.1:42362)
0806 02:47:47.161803 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0806 02:47:47.161931 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260806 02:47:47.163380 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:47.162911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:46093 (local address 127.0.0.1:42372)
0806 02:47:47.163096 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 02:47:47.163193 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260806 02:47:47.222177 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 13522
I20260806 02:47:47.224263 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:47.223671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:46093 (local address 127.0.0.1:42382)
0806 02:47:47.223959 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0806 02:47:47.224065 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260806 02:47:47.225929 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:47.225410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:46093 (local address 127.0.0.1:42396)
0806 02:47:47.225657 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0806 02:47:47.225746 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260806 02:47:47.243038 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 13665
Aug 06 02:47:47 dist-test-slave-8tdl krb5kdc[12827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.67@KRBTEST.COM
I20260806 02:47:47.264094 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:47.243843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54888 (local address 127.1.210.67:45197)
0806 02:47:47.244104 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0806 02:47:47.244110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:47:47.244131 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:47:47.244288 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:47:47.244292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:47:47.244381 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:47:47.244511 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:47:47.245950 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:47.246885 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:47.249037 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:47.249757 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:47.249866 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:47.255810 (+ 5944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:47:47.255824 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:47:47.255838 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:47:47.255887 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:47:47.260276 (+ 4389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:47:47.261387 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:47:47.261398 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:47:47.261409 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:47:47.261618 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:47:47.262153 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:47:47.262158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:47:47.262162 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:47:47.262753 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:47:47.263079 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0806 02:47:47.263435 (+ 356us) server_negotiation.cc:299] Negotiation successful
0806 02:47:47.263693 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 02:47:47.265069 13788 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:47:47.306599 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:47.306124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:46093 (local address 127.0.0.1:42398)
0806 02:47:47.306345 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0806 02:47:47.306431 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260806 02:47:47.307695 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:47.307294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:46093 (local address 127.0.0.1:42404)
0806 02:47:47.307478 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0806 02:47:47.307562 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260806 02:47:47.391265 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 13665
I20260806 02:47:47.412619 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 13271
I20260806 02:47:47.413874 13333 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:47:47.534905 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:47.533910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:46093 (local address 127.0.0.1:42414)
0806 02:47:47.534382 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0806 02:47:47.534518 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260806 02:47:47.535091 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 13271
I20260806 02:47:47.565023 1865 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 02:47:47.905175 1865 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 02:47:47.905529 1865 mini_ranger.cc:66] Stopping Ranger...
I20260806 02:47:48.251875 1865 mini_ranger.cc:68] Stopped Ranger
2026-08-06 02:47:48.252 UTC [12857] LOG: received smart shutdown request
2026-08-06 02:47:48.256 UTC [12857] LOG: background worker "logical replication launcher" (PID 12866) exited with exit code 1
2026-08-06 02:47:48.257 UTC [12861] LOG: shutting down
2026-08-06 02:47:48.476 UTC [12857] LOG: database system is shut down
2026-08-06T02:47:48Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (56851 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:47:48 dist-test-slave-8tdl krb5kdc[13853](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:47:48 dist-test-slave-8tdl krb5kdc[13853](info): set up 2 sockets
Aug 06 02:47:48 dist-test-slave-8tdl krb5kdc[13853](info): commencing operation
krb5kdc: starting...
W20260806 02:47:49.795437 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.137s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:47:49 dist-test-slave-8tdl krb5kdc[13853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:47:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:47:49Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.210.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.210.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.210.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/rangeradmin_127.1.210.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.210.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/rangeradmin_127.1.210.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.210.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.210.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.210.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/rangerlookup_127.1.210.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.210.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/rangerlookup_127.1.210.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.210.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.210.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.210.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/HTTP_127.1.210.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.210.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/HTTP_127.1.210.84@KRBTEST.COM.keytab.
I20260806 02:47:50.194983 1865 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task5Fb5fs/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/postgres -l logfile start
2026-08-06 02:47:53.073 UTC [13883] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 02:47:53.073 UTC [13883] LOG: listening on IPv4 address "127.1.210.84", port 52945
2026-08-06 02:47:53.077 UTC [13883] LOG: listening on Unix socket "/tmp/.s.PGSQL.52945"
2026-08-06 02:47:53.103 UTC [13886] LOG: database system was shut down at 2026-08-06 02:47:51 UTC
2026-08-06 02:47:53.108 UTC [13883] LOG: database system is ready to accept connections
I20260806 02:47:54.102125 1865 mini_postgres.cc:96] Postgres bound to 52945
2026-08-06 02:47:54.123 UTC [13894] FATAL: database "slave" does not exist
127.1.210.84:52945 - accepting connections
I20260806 02:47:54.124442 1865 mini_ranger.cc:161] Starting Ranger...
I20260806 02:47:54.154919 1865 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 02:47:54.736853 1865 mini_ranger.cc:87] Created ranger Postgres database
I20260806 02:47:54.736994 1865 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ranger-admin
2026-08-06 02:47:55,047 [I] DB FLAVOR :POSTGRES
2026-08-06 02:47:55,048 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 02:47:55,048 [I] Checking connection..
2026-08-06 02:47:55,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:52945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 02:47:55,327 [I] Checking connection passed.
2026-08-06 02:47:55,328 [I] --------- Verifying version history table ---------
2026-08-06 02:47:55,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:52945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 02:47:55,603 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 02:47:55,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:52945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 02:47:55,870 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 02:47:55,870 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 02:47:55,870 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:52945/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 02:47:56.150 UTC [13959] WARNING: there is no transaction in progress
2026-08-06 02:47:56,157 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 02:47:56,158 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:52945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 02:47:56,431 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 02:47:56,431 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 02:47:56,431 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:52945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 02:47:56,693 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:52945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 02:47:56,962 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:52945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8tdl.c.gcp-upstream.internal','N') ;"
2026-08-06 02:47:57,204 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:52945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 02:47:57,471 [I] Table x_portal_user does not exist in database ranger
2026-08-06 02:47:57,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:52945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 02:47:57,726 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 02:47:57,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:52945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 02:47:57,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:52945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 02:47:58,243 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 02:47:58,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:52945/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 02:47:59.364 UTC [14068] WARNING: there is no transaction in progress
2026-08-06 02:47:59.382 UTC [14068] WARNING: there is no transaction in progress
2026-08-06 02:47:59.401 UTC [14068] WARNING: there is no transaction in progress
2026-08-06 02:47:59.420 UTC [14068] WARNING: there is no transaction in progress
2026-08-06 02:47:59.438 UTC [14068] WARNING: there is no transaction in progress
2026-08-06 02:47:59.459 UTC [14068] WARNING: there is no transaction in progress
2026-08-06 02:47:59.523 UTC [14068] WARNING: there is no transaction in progress
2026-08-06 02:47:59.531 UTC [14068] WARNING: there is no transaction in progress
2026-08-06 02:47:59.548 UTC [14068] WARNING: there is no transaction in progress
2026-08-06 02:47:59.560 UTC [14068] WARNING: there is no transaction in progress
2026-08-06 02:47:59.571 UTC [14068] WARNING: there is no transaction in progress
2026-08-06 02:47:59.587 UTC [14068] WARNING: there is no transaction in progress
2026-08-06 02:47:59.600 UTC [14068] WARNING: there is no transaction in progress
2026-08-06 02:47:59.608 UTC [14068] WARNING: there is no transaction in progress
2026-08-06 02:47:59.618 UTC [14068] WARNING: there is no transaction in progress
2026-08-06 02:47:59.626 UTC [14068] WARNING: there is no transaction in progress
2026-08-06 02:48:00,447 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 02:48:00,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:52945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 02:48:00,721 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 02:48:00,722 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:52945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 02:48:01,009 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 02:48:01,010 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:52945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 02:48:01,259 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 02:48:01,259 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:52945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 02:48:01,536 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:52945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 02:48:01,796 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:52945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8tdl.c.gcp-upstream.internal';"
2026-08-06 02:48:02,032 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 02:48:02,032 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:52945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 02:48:02,284 [I] DB_PATCHES have already been applied
I20260806 02:48:02.291220 1865 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ranger-admin:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task5Fb5fs/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 02:48:02.291312 1865 mini_ranger.cc:193] Using host: 127.1.210.84
I20260806 02:48:02.310045 1865 mini_ranger.cc:227] Ranger admin URL: http://127.1.210.84:36295
Aug 06, 2026 2:48:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 2:48:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 2:48:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 2:48:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 2:48:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 2:48:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 2:48:03 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 02:48:03.493398 1865 mini_ranger.cc:160] Time spent starting Ranger: real 9.369s user 0.000s sys 0.054s
Aug 06, 2026 2:48:26 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 2:48:26 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Aug 06, 2026 2:48:26 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 2:48:26 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 2:48:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 2:48:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 2:48:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 2:48:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 2:48:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 2:48:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 2:48:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 2:48:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 2:48:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 2:48:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 2:48:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 2:48:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 2:48:26 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 2:48:27 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 02:48:28.468466 1865 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.210.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.210.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.210.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/rangerkms_127.1.210.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.210.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/rangerkms_127.1.210.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.210.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.210.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.210.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/HTTP_127.1.210.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.210.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/HTTP_127.1.210.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 02:48:28 dist-test-slave-8tdl krb5kdc[13853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984508, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 02:48:28.678339 1865 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 02:48:28.703918 1865 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 02:48:29.931620 1865 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 02:48:29.931756 1865 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ranger-kms
I20260806 02:48:29.931811 1865 mini_ranger_kms.cc:227] Using postgres at 127.1.210.84:52945
2026-08-06 02:48:30,325 [I] DB FLAVOR :POSTGRES
2026-08-06 02:48:30,325 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 02:48:30,326 [I] Checking connection
2026-08-06 02:48:30,326 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:52945/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 02:48:30,587 [I] connection success
2026-08-06 02:48:30,587 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 02:48:30,587 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 02:48:30,588 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:52945/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 02:48:30,850 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 02:48:30,851 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 02:48:30,851 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 02:48:30,851 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.210.84:52945/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 02:48:31,132 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 02:48:31.725835 1865 mini_ranger_kms.cc:310] Created kms service
I20260806 02:48:31.952690 1865 mini_ranger_kms.cc:326] Created kudu user
I20260806 02:48:32.000312 1865 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 02:48:32 dist-test-slave-8tdl krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984508, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.210.84@KRBTEST.COM
I20260806 02:48:32.443250 1865 mini_ranger_kms.cc:384] Added ranger policy
I20260806 02:48:32.443607 1865 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ranger-kms:/tmp/dist-test-task5Fb5fs/build/asan/bin/postgresql.jar:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task5Fb5fs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 02:48:32.443670 1865 mini_ranger_kms.cc:240] Using host: 127.1.210.84
I20260806 02:48:32.463833 1865 mini_ranger_kms.cc:276] Ranger KMS PID: 14250
I20260806 02:48:32.464015 1865 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.210.84:33933
Aug 06, 2026 2:48:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 2:48:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 2:48:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 2:48:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 2:48:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 2:48:34 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260806 02:48:34.870786 1865 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.192s user 0.090s sys 0.331s
I20260806 02:48:34.871085 1865 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 02:48:34.871282 1865 mini_ranger_kms.cc:401] 127.1.210.84:33933/kms/v1/keys
Aug 06 02:48:34 dist-test-slave-8tdl krb5kdc[13853](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785984514, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.210.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 06 02:48:35 dist-test-slave-8tdl krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984514, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.210.84@KRBTEST.COM for HTTP/127.1.210.84@KRBTEST.COM
Aug 06 02:48:35 dist-test-slave-8tdl krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984514, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.210.84@KRBTEST.COM for HTTP/127.1.210.84@KRBTEST.COM
Aug 06 02:48:38 dist-test-slave-8tdl krb5kdc[13853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
I20260806 02:48:38.524863 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:43881
--webserver_interface=127.1.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:34445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:43881
--ranger_config_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.210.84:33933/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:48:38.889815 14300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:38.890219 14300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:38.890403 14300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:38.898168 14300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:48:38.898337 14300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:38.898453 14300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:38.898552 14300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:48:38.898643 14300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:48:38.909274 14300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:34445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.210.84:33933/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:43881
--ranger_config_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:43881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:48:38.910785 14300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:38.912248 14300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:38.926067 14308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:38.926188 14305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:38.926165 14306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:38.926541 14300 server_base.cc:1034] running on GCE node
I20260806 02:48:38.927762 14300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:38.929309 14300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:38.930554 14300 hybrid_clock.cc:648] HybridClock initialized: now 1785984518930510 us; error 39 us; skew 500 ppm
I20260806 02:48:38.930974 14300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:48:38 dist-test-slave-8tdl krb5kdc[13853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984518, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:38.934405 14300 init.cc:377] Logged in from keytab as kudu/127.1.210.126@KRBTEST.COM (short username kudu)
I20260806 02:48:38.936082 14300 webserver.cc:460] Webserver started at http://127.1.210.126:35453/ using document root <none> and password file <none>
I20260806 02:48:38.936754 14300 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:38.936856 14300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:38.937122 14300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:48:38 dist-test-slave-8tdl krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984518, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for HTTP/127.1.210.84@KRBTEST.COM
I20260806 02:48:39.006659 14300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "491706754f864fd3a7bddd91f4ebaf83"
format_stamp: "Formatted at 2026-08-06 02:48:39 on dist-test-slave-8tdl"
server_key: "3e107569bc79c1b556fc600b1baea366"
server_key_iv: "ec0e53d7a86ea2971d128a8aa21a755b"
server_key_version: "kuduclusterkey@0"
I20260806 02:48:39.007856 14300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "491706754f864fd3a7bddd91f4ebaf83"
format_stamp: "Formatted at 2026-08-06 02:48:39 on dist-test-slave-8tdl"
server_key: "3e107569bc79c1b556fc600b1baea366"
server_key_iv: "ec0e53d7a86ea2971d128a8aa21a755b"
server_key_version: "kuduclusterkey@0"
I20260806 02:48:39.014290 14300 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260806 02:48:39.032130 14315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:39.033867 14300 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260806 02:48:39.034092 14300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "491706754f864fd3a7bddd91f4ebaf83"
format_stamp: "Formatted at 2026-08-06 02:48:39 on dist-test-slave-8tdl"
server_key: "3e107569bc79c1b556fc600b1baea366"
server_key_iv: "ec0e53d7a86ea2971d128a8aa21a755b"
server_key_version: "kuduclusterkey@0"
I20260806 02:48:39.034350 14300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:39.048931 14300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:39.053524 14300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:39.053918 14300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:39.075124 14300 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:43881
I20260806 02:48:39.075131 14366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:43881 every 8 connection(s)
I20260806 02:48:39.076628 14300 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:48:39.078792 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 14300
I20260806 02:48:39.079037 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:48:39.079916 1865 external_mini_cluster.cc:1428] Setting key 143a5f439653eb9f7cd64a213184894c
I20260806 02:48:39.081348 14367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:48:39.098995 14367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83: Bootstrap starting.
I20260806 02:48:39.103113 14367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83: Neither blocks nor log segments found. Creating new log.
I20260806 02:48:39.104566 14367 log.cc:824] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:39.109012 14367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83: No bootstrap required, opened a new log
Aug 06 02:48:39 dist-test-slave-8tdl krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:48:39.122262 14367 raft_consensus.cc:348] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491706754f864fd3a7bddd91f4ebaf83" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 43881 } }
I20260806 02:48:39.122918 14367 raft_consensus.cc:374] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:48:39.123104 14367 raft_consensus.cc:729] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491706754f864fd3a7bddd91f4ebaf83, State: Initialized, Role: FOLLOWER
I20260806 02:48:39.123868 14367 consensus_queue.cc:260] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491706754f864fd3a7bddd91f4ebaf83" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 43881 } }
I20260806 02:48:39.124150 14367 raft_consensus.cc:388] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:48:39.124336 14367 raft_consensus.cc:482] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:48:39.124346 14370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:39.082234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39490 (local address 127.1.210.126:43881)
0806 02:48:39.095378 (+ 13144us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:39.095397 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:39.095463 (+ 66us) server_negotiation.cc:407] Connection header received
0806 02:48:39.098070 (+ 2607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:39.098107 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:39.098730 (+ 623us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:39.099227 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:39.101212 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:39.103417 (+ 2205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:39.105595 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:39.106833 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:39.107064 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:39.113263 (+ 6199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:39.113277 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:39.113311 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:39.113378 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:39.118242 (+ 4864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:39.119740 (+ 1498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:39.119752 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:39.119762 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:39.120020 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:39.120583 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:39.120589 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:39.120593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:39.121459 (+ 866us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:39.121820 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:39.122366 (+ 546us) server_negotiation.cc:299] Negotiation successful
0806 02:48:39.122890 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12871,"thread_start_us":12616,"threads_started":1}
I20260806 02:48:39.124590 14367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:48:39.128139 14367 raft_consensus.cc:504] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491706754f864fd3a7bddd91f4ebaf83" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 43881 } }
I20260806 02:48:39.128777 14367 leader_election.cc:302] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 491706754f864fd3a7bddd91f4ebaf83; no voters:
I20260806 02:48:39.129531 14367 leader_election.cc:288] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:48:39.129666 14372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:48:39.131592 14372 raft_consensus.cc:686] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83 [term 1 LEADER]: Becoming Leader. State: Replica: 491706754f864fd3a7bddd91f4ebaf83, State: Running, Role: LEADER
I20260806 02:48:39.132373 14372 consensus_queue.cc:237] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491706754f864fd3a7bddd91f4ebaf83" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 43881 } }
I20260806 02:48:39.133275 14367 sys_catalog.cc:564] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:48:39.136485 14374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 491706754f864fd3a7bddd91f4ebaf83. Latest consensus state: current_term: 1 leader_uuid: "491706754f864fd3a7bddd91f4ebaf83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491706754f864fd3a7bddd91f4ebaf83" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 43881 } } }
I20260806 02:48:39.136801 14374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83 [sys.catalog]: This master's current role is: LEADER
I20260806 02:48:39.135989 14373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "491706754f864fd3a7bddd91f4ebaf83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491706754f864fd3a7bddd91f4ebaf83" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 43881 } } }
I20260806 02:48:39.137466 14373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83 [sys.catalog]: This master's current role is: LEADER
I20260806 02:48:39.148404 14379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:48:39.154958 14379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:48:39.155817 14367 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 02:48:39.175536 14379 catalog_manager.cc:1347] Generated new cluster ID: 620b0522dc094da98fc32a26f7d137d9
I20260806 02:48:39.175839 14379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:48:39.189157 14379 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:48:39.190658 14379 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:48:39.200131 14379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83: Generated new TSK 0
I20260806 02:48:39.201073 14379 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab.
Entry for principal kudu/127.1.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab.
I20260806 02:48:40.047670 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.210.84:33933/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.65:0
--local_ip_for_outbound_sockets=127.1.210.65
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:43881
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:34445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:48:40 dist-test-slave-8tdl krb5kdc[13853](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785984520, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 02:48:40.539222 14414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:40.539772 14414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:40.539958 14414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:40.553563 14414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:40.553822 14414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:40.554039 14414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.65
I20260806 02:48:40.566494 14414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:34445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.210.84:33933/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:43881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.65
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:48:40.568058 14414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:40.569484 14414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:40.590622 14424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:40.600478 14425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:40.600729 14427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:40.601718 14414 server_base.cc:1034] running on GCE node
I20260806 02:48:40.602777 14414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:40.604568 14414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:40.605876 14414 hybrid_clock.cc:648] HybridClock initialized: now 1785984520605807 us; error 69 us; skew 500 ppm
I20260806 02:48:40.606287 14414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:48:40 dist-test-slave-8tdl krb5kdc[13853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984520, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:40.609845 14414 init.cc:377] Logged in from keytab as kudu/127.1.210.65@KRBTEST.COM (short username kudu)
I20260806 02:48:40.611907 14414 webserver.cc:460] Webserver started at http://127.1.210.65:33575/ using document root <none> and password file <none>
I20260806 02:48:40.612993 14414 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:40.613195 14414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:40.613762 14414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:48:40 dist-test-slave-8tdl krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984520, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for HTTP/127.1.210.84@KRBTEST.COM
I20260806 02:48:40.633785 14414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-0/data/instance:
uuid: "9aef7f7c5bc7414f8e0df3d807c556db"
format_stamp: "Formatted at 2026-08-06 02:48:40 on dist-test-slave-8tdl"
server_key: "fbdc6f4e04e5ed9ab711591302c7b2f3"
server_key_iv: "eb3f4b351271fa7e27c18030d5cc862f"
server_key_version: "kuduclusterkey@0"
I20260806 02:48:40.635097 14414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance:
uuid: "9aef7f7c5bc7414f8e0df3d807c556db"
format_stamp: "Formatted at 2026-08-06 02:48:40 on dist-test-slave-8tdl"
server_key: "fbdc6f4e04e5ed9ab711591302c7b2f3"
server_key_iv: "eb3f4b351271fa7e27c18030d5cc862f"
server_key_version: "kuduclusterkey@0"
I20260806 02:48:40.662523 14414 fs_manager.cc:691] Time spent creating directory manager: real 0.027s user 0.013s sys 0.012s
I20260806 02:48:40.695503 14434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:40.697809 14414 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.007s sys 0.004s
I20260806 02:48:40.698091 14414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-0/wal
uuid: "9aef7f7c5bc7414f8e0df3d807c556db"
format_stamp: "Formatted at 2026-08-06 02:48:40 on dist-test-slave-8tdl"
server_key: "fbdc6f4e04e5ed9ab711591302c7b2f3"
server_key_iv: "eb3f4b351271fa7e27c18030d5cc862f"
server_key_version: "kuduclusterkey@0"
I20260806 02:48:40.698446 14414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:40.720469 14414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:40.725478 14414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:40.725832 14414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:40.730434 14414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:48:40.732218 14414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:48:40.732360 14414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:40.732540 14414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:48:40.732618 14414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:48:40 dist-test-slave-8tdl krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984520, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for HTTP/127.1.210.84@KRBTEST.COM
I20260806 02:48:40.899186 14414 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.65:36801
I20260806 02:48:40.899230 14546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.65:36801 every 8 connection(s)
I20260806 02:48:40.901021 14414 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
I20260806 02:48:40.905668 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 14414
I20260806 02:48:40.905885 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance
I20260806 02:48:40.906700 1865 external_mini_cluster.cc:1428] Setting key d1f645642ecfc7b09d3b733928ed98d9
Aug 06 02:48:40 dist-test-slave-8tdl krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984520, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.210.66@KRBTEST.COM; defaulting to no policy
I20260806 02:48:40.973865 14550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:40.907008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:42329 (local address 127.1.210.126:43881)
0806 02:48:40.939447 (+ 32439us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:40.939459 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:40.941840 (+ 2381us) server_negotiation.cc:407] Connection header received
0806 02:48:40.947384 (+ 5544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:40.947394 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:40.947512 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:40.947814 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:40.951674 (+ 3860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:40.953053 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:40.955940 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:40.956677 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:40.956791 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:40.961984 (+ 5193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:40.961992 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:40.961998 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:40.962048 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:40.966151 (+ 4103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:40.967342 (+ 1191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:40.967349 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:40.967353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:40.967527 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:40.970887 (+ 3360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:40.970896 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:40.970901 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:40.971412 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:40.971613 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:40.972998 (+ 1385us) server_negotiation.cc:299] Negotiation successful
0806 02:48:40.973343 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32018,"thread_start_us":32155,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.66@KRBTEST.COM" created.
I20260806 02:48:40.976775 14549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:40.904778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:43881 (local address 127.1.210.65:42329)
0806 02:48:40.941207 (+ 36429us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:40.941270 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:40.944112 (+ 2842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:40.948158 (+ 4046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:40.948174 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:40.949073 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:40.950207 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:40.950241 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:40.953299 (+ 3058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:40.953306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:40.955683 (+ 2377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:40.955703 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:40.957224 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:40.957231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:40.957557 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:40.958555 (+ 998us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:40.958589 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:40.961639 (+ 3050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:40.966397 (+ 4758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:40.966408 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:40.966433 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:40.967172 (+ 739us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:40.967689 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:40.967694 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:40.967698 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:40.967991 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:40.971752 (+ 3761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:40.971763 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:40.972389 (+ 626us) client_negotiation.cc:769] Sending connection context
0806 02:48:40.974289 (+ 1900us) client_negotiation.cc:240] Negotiation successful
0806 02:48:40.975072 (+ 783us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36040,"thread_start_us":35775,"threads_started":1}
I20260806 02:48:40.979951 14547 heartbeater.cc:347] Connected to a master server at 127.1.210.126:43881
I20260806 02:48:40.980357 14547 heartbeater.cc:464] Registering TS with master...
I20260806 02:48:40.981391 14547 heartbeater.cc:511] Master 127.1.210.126:43881 requested a full tablet report, sending...
I20260806 02:48:40.984453 14332 ts_manager.cc:194] Registered new tserver with Master: 9aef7f7c5bc7414f8e0df3d807c556db (127.1.210.65:36801)
I20260806 02:48:40.987254 14332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.65@KRBTEST.COM'} at 127.1.210.65:42329
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab.
Entry for principal kudu/127.1.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab.
W20260806 02:48:41.018834 1865 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.1.210.66: real 0.108s user 0.008s sys 0.017s
I20260806 02:48:41.019999 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.210.84:33933/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.66:0
--local_ip_for_outbound_sockets=127.1.210.66
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:43881
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:34445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:48:41 dist-test-slave-8tdl krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984520, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for HTTP/127.1.210.84@KRBTEST.COM
W20260806 02:48:41.504030 14553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:41.504325 14553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:41.504421 14553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:41.511875 14553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:41.512060 14553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:41.512192 14553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.66
I20260806 02:48:41.522181 14553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:34445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.210.84:33933/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:43881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.66
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:48:41.523581 14553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:41.524832 14553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:41.538734 14570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:41.538772 14567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:41.538822 14568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:41.544718 14553 server_base.cc:1034] running on GCE node
I20260806 02:48:41.545419 14553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:41.546507 14553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:41.547770 14553 hybrid_clock.cc:648] HybridClock initialized: now 1785984521547725 us; error 80 us; skew 500 ppm
I20260806 02:48:41.548147 14553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:48:41 dist-test-slave-8tdl krb5kdc[13853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984521, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:41.552044 14553 init.cc:377] Logged in from keytab as kudu/127.1.210.66@KRBTEST.COM (short username kudu)
I20260806 02:48:41.553812 14553 webserver.cc:460] Webserver started at http://127.1.210.66:37969/ using document root <none> and password file <none>
I20260806 02:48:41.554551 14553 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:41.554663 14553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:41.555042 14553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:48:41 dist-test-slave-8tdl krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984521, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for HTTP/127.1.210.84@KRBTEST.COM
I20260806 02:48:41.573251 14553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-1/data/instance:
uuid: "4044f79d135f48b2ad8d2253454e625c"
format_stamp: "Formatted at 2026-08-06 02:48:41 on dist-test-slave-8tdl"
server_key: "7d1f684cedefd87f3c9dc89d9bf28904"
server_key_iv: "489073c8238ff52c45f907ca9c7c9c61"
server_key_version: "kuduclusterkey@0"
I20260806 02:48:41.574613 14553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance:
uuid: "4044f79d135f48b2ad8d2253454e625c"
format_stamp: "Formatted at 2026-08-06 02:48:41 on dist-test-slave-8tdl"
server_key: "7d1f684cedefd87f3c9dc89d9bf28904"
server_key_iv: "489073c8238ff52c45f907ca9c7c9c61"
server_key_version: "kuduclusterkey@0"
I20260806 02:48:41.580332 14553 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260806 02:48:41.599079 14577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:41.600754 14553 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:48:41.600997 14553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-1/wal
uuid: "4044f79d135f48b2ad8d2253454e625c"
format_stamp: "Formatted at 2026-08-06 02:48:41 on dist-test-slave-8tdl"
server_key: "7d1f684cedefd87f3c9dc89d9bf28904"
server_key_iv: "489073c8238ff52c45f907ca9c7c9c61"
server_key_version: "kuduclusterkey@0"
I20260806 02:48:41.601256 14553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:41.630895 14553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:41.635902 14553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:41.636197 14553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:41.637285 14553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:48:41.639114 14553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:48:41.639195 14553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:41.639346 14553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:48:41.639423 14553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:41.679199 14553 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.66:36883
I20260806 02:48:41.679255 14689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.66:36883 every 8 connection(s)
I20260806 02:48:41.680684 14553 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
I20260806 02:48:41.686257 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 14553
I20260806 02:48:41.686479 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance
I20260806 02:48:41.687098 1865 external_mini_cluster.cc:1428] Setting key 57354266c7c5f25516b7e2b7b1d8a32e
Aug 06 02:48:41 dist-test-slave-8tdl krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984521, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:48:41.707618 14692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:41.683784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:41871 (local address 127.1.210.126:43881)
0806 02:48:41.684369 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:41.684378 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:41.685512 (+ 1134us) server_negotiation.cc:407] Connection header received
0806 02:48:41.688200 (+ 2688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:41.688207 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:41.688320 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:41.688524 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:41.690607 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:41.691565 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:41.694365 (+ 2800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:41.694926 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:41.695048 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:41.699799 (+ 4751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:41.699809 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:41.699817 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:41.699869 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:41.703273 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:41.704447 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:41.704452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:41.704454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:41.704609 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:41.705286 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:41.705291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:41.705293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:41.705707 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:41.705861 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:41.707078 (+ 1217us) server_negotiation.cc:299] Negotiation successful
0806 02:48:41.707342 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":330,"threads_started":1}
I20260806 02:48:41.709231 14693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:41.684525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:43881 (local address 127.1.210.66:41871)
0806 02:48:41.685329 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:41.685371 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:41.687839 (+ 2468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:41.688795 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:41.688805 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:41.689450 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:41.690370 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:41.690405 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:41.691759 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:41.691768 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:41.694155 (+ 2387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:41.694176 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:41.695193 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:41.695202 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:41.695594 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:41.696631 (+ 1037us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:41.696679 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:41.699515 (+ 2836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:41.703483 (+ 3968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:41.703498 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:41.703523 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:41.704290 (+ 767us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:41.704771 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:41.704779 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:41.704787 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:41.705139 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:41.705873 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:41.705887 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:41.706436 (+ 549us) client_negotiation.cc:769] Sending connection context
0806 02:48:41.706844 (+ 408us) client_negotiation.cc:240] Negotiation successful
0806 02:48:41.707521 (+ 677us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":319,"threads_started":1}
I20260806 02:48:41.711062 14690 heartbeater.cc:347] Connected to a master server at 127.1.210.126:43881
I20260806 02:48:41.711463 14690 heartbeater.cc:464] Registering TS with master...
I20260806 02:48:41.712239 14690 heartbeater.cc:511] Master 127.1.210.126:43881 requested a full tablet report, sending...
I20260806 02:48:41.713932 14332 ts_manager.cc:194] Registered new tserver with Master: 4044f79d135f48b2ad8d2253454e625c (127.1.210.66:36883)
I20260806 02:48:41.714848 14332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.66@KRBTEST.COM'} at 127.1.210.66:41871
WARNING: no policy specified for kudu/127.1.210.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab.
Entry for principal kudu/127.1.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab.
I20260806 02:48:41.759282 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.210.84:33933/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.67:0
--local_ip_for_outbound_sockets=127.1.210.67
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:43881
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:34445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:48:41.992177 14547 heartbeater.cc:503] Master 127.1.210.126:43881 was elected leader, sending a full tablet report...
W20260806 02:48:42.128283 14696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:42.128643 14696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:42.128815 14696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:42.136878 14696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:42.137054 14696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:42.137238 14696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.67
I20260806 02:48:42.147828 14696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:34445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.210.84:33933/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:43881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.67
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:48:42.149398 14696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:42.151271 14696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:42.165266 14702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:42.165335 14703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:42.165611 14705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:42.166482 14696 server_base.cc:1034] running on GCE node
I20260806 02:48:42.167250 14696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:42.168334 14696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:42.169603 14696 hybrid_clock.cc:648] HybridClock initialized: now 1785984522169555 us; error 45 us; skew 500 ppm
I20260806 02:48:42.169955 14696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:48:42 dist-test-slave-8tdl krb5kdc[13853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984522, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:42.173413 14696 init.cc:377] Logged in from keytab as kudu/127.1.210.67@KRBTEST.COM (short username kudu)
I20260806 02:48:42.175180 14696 webserver.cc:460] Webserver started at http://127.1.210.67:36001/ using document root <none> and password file <none>
I20260806 02:48:42.175927 14696 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:42.176085 14696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:42.176410 14696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:48:42 dist-test-slave-8tdl krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984522, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for HTTP/127.1.210.84@KRBTEST.COM
I20260806 02:48:42.195746 14696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-2/data/instance:
uuid: "c0cf376deec64958b6a08c98ee5e67f7"
format_stamp: "Formatted at 2026-08-06 02:48:42 on dist-test-slave-8tdl"
server_key: "71528d723ce60dbb5a3226c3668be73c"
server_key_iv: "6beafdd5b8c8c06d023a06ab65ee52b7"
server_key_version: "kuduclusterkey@0"
I20260806 02:48:42.196748 14696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance:
uuid: "c0cf376deec64958b6a08c98ee5e67f7"
format_stamp: "Formatted at 2026-08-06 02:48:42 on dist-test-slave-8tdl"
server_key: "71528d723ce60dbb5a3226c3668be73c"
server_key_iv: "6beafdd5b8c8c06d023a06ab65ee52b7"
server_key_version: "kuduclusterkey@0"
I20260806 02:48:42.203150 14696 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 02:48:42.222491 14712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:42.224182 14696 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:48:42.224433 14696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-2/wal
uuid: "c0cf376deec64958b6a08c98ee5e67f7"
format_stamp: "Formatted at 2026-08-06 02:48:42 on dist-test-slave-8tdl"
server_key: "71528d723ce60dbb5a3226c3668be73c"
server_key_iv: "6beafdd5b8c8c06d023a06ab65ee52b7"
server_key_version: "kuduclusterkey@0"
I20260806 02:48:42.224689 14696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:42.237278 14696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:42.241628 14696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:42.241945 14696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:42.242960 14696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:48:42.244583 14696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:48:42.244678 14696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:42.244812 14696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:48:42.244894 14696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:42.280905 14696 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.67:33129
I20260806 02:48:42.280966 14825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.67:33129 every 8 connection(s)
I20260806 02:48:42.282265 14696 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
I20260806 02:48:42.290724 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 14696
I20260806 02:48:42.290923 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance
I20260806 02:48:42.291406 1865 external_mini_cluster.cc:1428] Setting key 5b78a75816cc279170180ce94ca1cd16
Aug 06 02:48:42 dist-test-slave-8tdl krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984522, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:48:42.306211 14828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:42.285127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:41359 (local address 127.1.210.126:43881)
0806 02:48:42.285650 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:42.285657 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:42.286715 (+ 1058us) server_negotiation.cc:407] Connection header received
0806 02:48:42.288516 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:42.288521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:42.288615 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:42.288733 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:42.290430 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:42.291362 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:42.293656 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:42.294413 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:42.294609 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:42.298774 (+ 4165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:42.298780 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:42.298784 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:42.298823 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:42.302292 (+ 3469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:42.303338 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:42.303344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:42.303348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:42.303503 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:42.304007 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:42.304012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:42.304016 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:42.304450 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:42.304618 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:42.305494 (+ 876us) server_negotiation.cc:299] Negotiation successful
0806 02:48:42.305780 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":230,"threads_started":1}
I20260806 02:48:42.307498 14829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:42.285748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:43881 (local address 127.1.210.67:41359)
0806 02:48:42.286500 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:42.286541 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:42.288237 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:42.288978 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:42.288987 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:42.289446 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:42.290116 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:42.290137 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:42.291549 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:42.291555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:42.293461 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:42.293482 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:42.294795 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:42.294806 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:42.295224 (+ 418us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:42.296096 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:42.296122 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:42.298445 (+ 2323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:42.302492 (+ 4047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:42.302504 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:42.302525 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:42.303196 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:42.303629 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:42.303634 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:42.303637 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:42.303870 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:42.304589 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:42.304598 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:42.305049 (+ 451us) client_negotiation.cc:769] Sending connection context
0806 02:48:42.305403 (+ 354us) client_negotiation.cc:240] Negotiation successful
0806 02:48:42.306061 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":318,"threads_started":1}
I20260806 02:48:42.309053 14826 heartbeater.cc:347] Connected to a master server at 127.1.210.126:43881
I20260806 02:48:42.309398 14826 heartbeater.cc:464] Registering TS with master...
I20260806 02:48:42.310197 14826 heartbeater.cc:511] Master 127.1.210.126:43881 requested a full tablet report, sending...
I20260806 02:48:42.311836 14332 ts_manager.cc:194] Registered new tserver with Master: c0cf376deec64958b6a08c98ee5e67f7 (127.1.210.67:33129)
I20260806 02:48:42.312887 14332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.67@KRBTEST.COM'} at 127.1.210.67:41359
I20260806 02:48:42.321333 1865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:48:42.323441 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 14414
Aug 06 02:48:42 dist-test-slave-8tdl krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.65@KRBTEST.COM
I20260806 02:48:42.346952 14831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:42.324607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41500 (local address 127.1.210.65:36801)
0806 02:48:42.325305 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:42.325313 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:42.327358 (+ 2045us) server_negotiation.cc:407] Connection header received
0806 02:48:42.327557 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:42.327568 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:42.327843 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:42.328049 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:42.330117 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:42.331704 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:42.333486 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:42.334368 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:42.334505 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:42.340147 (+ 5642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:42.340171 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:42.340193 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:42.340242 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:42.343643 (+ 3401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:42.344623 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:42.344642 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:42.344644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:42.344802 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:42.345298 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:42.345303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:42.345305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:42.345711 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:42.346000 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:42.346301 (+ 301us) server_negotiation.cc:299] Negotiation successful
0806 02:48:42.346532 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":333,"threads_started":1}
I20260806 02:48:42.348497 14542 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:48:42.435930 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 14414
I20260806 02:48:42.455240 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 14553
Aug 06 02:48:42 dist-test-slave-8tdl krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.66@KRBTEST.COM
I20260806 02:48:42.480981 14833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:42.461210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39676 (local address 127.1.210.66:36883)
0806 02:48:42.461782 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:42.461790 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:42.461817 (+ 27us) server_negotiation.cc:407] Connection header received
0806 02:48:42.461936 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:42.461944 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:42.462197 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:42.462335 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:42.463779 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:42.465056 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:42.466588 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:42.467434 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:42.467562 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:42.472918 (+ 5356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:42.472940 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:42.472967 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:42.473058 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:42.477525 (+ 4467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:42.478568 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:42.478576 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:42.478579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:42.478810 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:42.479289 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:42.479293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:42.479295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:42.479699 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:42.479963 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:42.480302 (+ 339us) server_negotiation.cc:299] Negotiation successful
0806 02:48:42.480549 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":303,"threads_started":1}
I20260806 02:48:42.482436 14685 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:48:42.568099 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 14553
I20260806 02:48:42.584997 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 14696
Aug 06 02:48:42 dist-test-slave-8tdl krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.67@KRBTEST.COM
I20260806 02:48:42.604555 14835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:42.586398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36582 (local address 127.1.210.67:33129)
0806 02:48:42.587215 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:42.587223 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:42.587254 (+ 31us) server_negotiation.cc:407] Connection header received
0806 02:48:42.587319 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:42.587325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:42.587619 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:42.587827 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:42.589882 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:42.591057 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:42.592560 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:42.593126 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:42.593269 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:42.598392 (+ 5123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:42.598407 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:42.598423 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:42.598476 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:42.601404 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:42.602224 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:42.602233 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:42.602236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:42.602381 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:42.602802 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:42.602810 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:42.602819 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:42.603198 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:42.603538 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:42.603878 (+ 340us) server_negotiation.cc:299] Negotiation successful
0806 02:48:42.604181 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":455,"threads_started":1}
I20260806 02:48:42.606146 14821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:48:42.691915 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 14696
I20260806 02:48:42.708514 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 14300
I20260806 02:48:42.709905 14362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:48:42.820192 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 14300
I20260806 02:48:42.844053 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:43881
--webserver_interface=127.1.210.126
--webserver_port=35453
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:34445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:43881
--ranger_config_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.210.84:33933/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.210.84:33933/kms with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:48:43.200839 14838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:43.201223 14838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:43.201395 14838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:43.208938 14838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:48:43.209117 14838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:43.209232 14838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:43.209343 14838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:48:43.209429 14838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:48:43.220428 14838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:34445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.210.84:33933/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:43881
--ranger_config_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:43881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=35453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:48:43.221860 14838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:43.223381 14838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:43.236168 14843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:43.236344 14846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:43.236377 14844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:43.237525 14838 server_base.cc:1034] running on GCE node
I20260806 02:48:43.238463 14838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:43.240195 14838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:43.241436 14838 hybrid_clock.cc:648] HybridClock initialized: now 1785984523241354 us; error 69 us; skew 500 ppm
I20260806 02:48:43.241861 14838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:48:43 dist-test-slave-8tdl krb5kdc[13853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984523, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:43.250321 14838 init.cc:377] Logged in from keytab as kudu/127.1.210.126@KRBTEST.COM (short username kudu)
I20260806 02:48:43.252332 14838 webserver.cc:460] Webserver started at http://127.1.210.126:35453/ using document root <none> and password file <none>
I20260806 02:48:43.253104 14838 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:43.253290 14838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 02:48:43 dist-test-slave-8tdl krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984523, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for HTTP/127.1.210.84@KRBTEST.COM
I20260806 02:48:43.307360 14838 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:48:43.310911 14854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:43.312475 14838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:48:43.312698 14838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "491706754f864fd3a7bddd91f4ebaf83"
format_stamp: "Formatted at 2026-08-06 02:48:39 on dist-test-slave-8tdl"
server_key: "3e107569bc79c1b556fc600b1baea366"
server_key_iv: "ec0e53d7a86ea2971d128a8aa21a755b"
server_key_version: "kuduclusterkey@0"
I20260806 02:48:43.313720 14838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:43.349128 14838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:43.353595 14838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:43.354012 14838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:43.374818 14838 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:43881
I20260806 02:48:43.374943 14905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:43881 every 8 connection(s)
I20260806 02:48:43.376438 14838 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:48:43.383862 14906 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:48:43.383927 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 14838
I20260806 02:48:43.384867 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.210.84:33933/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.65:36801
--local_ip_for_outbound_sockets=127.1.210.65
--tserver_master_addrs=127.1.210.126:43881
--webserver_port=33575
--webserver_interface=127.1.210.65
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:34445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.210.84:33933/kms with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:48:43.387694 14906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83: Bootstrap starting.
I20260806 02:48:43.407647 14906 log.cc:824] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:43.418006 14906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:48:43.418728 14906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83: Bootstrap complete.
I20260806 02:48:43.428426 14906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491706754f864fd3a7bddd91f4ebaf83" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 43881 } }
I20260806 02:48:43.429474 14906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491706754f864fd3a7bddd91f4ebaf83, State: Initialized, Role: FOLLOWER
I20260806 02:48:43.430390 14906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491706754f864fd3a7bddd91f4ebaf83" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 43881 } }
I20260806 02:48:43.430651 14906 raft_consensus.cc:388] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:48:43.430797 14906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:48:43.430992 14906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83 [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:48:43.435060 14906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491706754f864fd3a7bddd91f4ebaf83" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 43881 } }
I20260806 02:48:43.435595 14906 leader_election.cc:302] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 491706754f864fd3a7bddd91f4ebaf83; no voters:
I20260806 02:48:43.436244 14906 leader_election.cc:288] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 02:48:43.436384 14910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 02:48:43.437810 14910 raft_consensus.cc:686] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83 [term 2 LEADER]: Becoming Leader. State: Replica: 491706754f864fd3a7bddd91f4ebaf83, State: Running, Role: LEADER
I20260806 02:48:43.438426 14910 consensus_queue.cc:237] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491706754f864fd3a7bddd91f4ebaf83" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 43881 } }
I20260806 02:48:43.438706 14906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:48:43.443552 14911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "491706754f864fd3a7bddd91f4ebaf83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491706754f864fd3a7bddd91f4ebaf83" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 43881 } } }
I20260806 02:48:43.444021 14911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83 [sys.catalog]: This master's current role is: LEADER
I20260806 02:48:43.444850 14912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 491706754f864fd3a7bddd91f4ebaf83. Latest consensus state: current_term: 2 leader_uuid: "491706754f864fd3a7bddd91f4ebaf83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491706754f864fd3a7bddd91f4ebaf83" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 43881 } } }
I20260806 02:48:43.445169 14912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83 [sys.catalog]: This master's current role is: LEADER
I20260806 02:48:43.449980 14920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:48:43.456175 14920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:48:43.457011 14906 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 02:48:43.457708 14920 catalog_manager.cc:1259] Loaded cluster ID: 620b0522dc094da98fc32a26f7d137d9
I20260806 02:48:43.457804 14920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:48:43.463406 14920 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:48:43.465485 14920 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83: Loaded TSK: 0
I20260806 02:48:43.467593 14920 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 02:48:43.866943 14908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:43.867384 14908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:43.867555 14908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:43.879576 14908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:43.879740 14908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:43.879961 14908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.65
I20260806 02:48:43.896416 14908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:34445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.210.84:33933/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.65:36801
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.65
--webserver_port=33575
--tserver_master_addrs=127.1.210.126:43881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.65
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:48:43.898346 14908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:43.900316 14908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:43.942962 14945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:43.962061 14948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:43.962641 14944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:43.963328 14908 server_base.cc:1034] running on GCE node
I20260806 02:48:43.964605 14908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:43.971503 14908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:43.973014 14908 hybrid_clock.cc:648] HybridClock initialized: now 1785984523972935 us; error 85 us; skew 500 ppm
I20260806 02:48:43.973505 14908 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:48:43 dist-test-slave-8tdl krb5kdc[13853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984523, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:43.978236 14908 init.cc:377] Logged in from keytab as kudu/127.1.210.65@KRBTEST.COM (short username kudu)
I20260806 02:48:44.011119 14908 webserver.cc:460] Webserver started at http://127.1.210.65:33575/ using document root <none> and password file <none>
I20260806 02:48:44.012297 14908 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:44.012544 14908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 02:48:44 dist-test-slave-8tdl krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984523, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for HTTP/127.1.210.84@KRBTEST.COM
I20260806 02:48:44.046006 14908 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.004s sys 0.003s
I20260806 02:48:44.050087 14956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:44.052364 14908 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260806 02:48:44.052587 14908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-0/wal
uuid: "9aef7f7c5bc7414f8e0df3d807c556db"
format_stamp: "Formatted at 2026-08-06 02:48:40 on dist-test-slave-8tdl"
server_key: "fbdc6f4e04e5ed9ab711591302c7b2f3"
server_key_iv: "eb3f4b351271fa7e27c18030d5cc862f"
server_key_version: "kuduclusterkey@0"
I20260806 02:48:44.054349 14908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:44.085590 14908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:44.091117 14908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:44.091423 14908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:44.092589 14908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:48:44.094416 14908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:48:44.094527 14908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:44.094631 14908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:48:44.094723 14908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260806 02:48:44.161185 14908 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.65:36801
I20260806 02:48:44.161232 15069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.65:36801 every 8 connection(s)
I20260806 02:48:44.162873 14908 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
I20260806 02:48:44.174232 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 14908
I20260806 02:48:44.175212 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.210.84:33933/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.66:36883
--local_ip_for_outbound_sockets=127.1.210.66
--tserver_master_addrs=127.1.210.126:43881
--webserver_port=37969
--webserver_interface=127.1.210.66
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:34445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.210.84:33933/kms with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:48:44 dist-test-slave-8tdl krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984523, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:48:44.219262 15072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:44.166966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:51583 (local address 127.1.210.126:43881)
0806 02:48:44.182289 (+ 15323us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:44.182322 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:44.185370 (+ 3048us) server_negotiation.cc:407] Connection header received
0806 02:48:44.188526 (+ 3156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:44.188553 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:44.189107 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:44.189546 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:44.191760 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:44.194074 (+ 2314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:44.199660 (+ 5586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:44.201103 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:44.201364 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:44.206754 (+ 5390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:44.206772 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:44.206801 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:44.206880 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:44.211534 (+ 4654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:44.212728 (+ 1194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:44.212740 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:44.212750 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:44.213006 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:44.213592 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:44.213597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:44.213601 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:44.214316 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:44.214518 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:44.217181 (+ 2663us) server_negotiation.cc:299] Negotiation successful
0806 02:48:44.217648 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14846,"thread_start_us":15249,"threads_started":1}
I20260806 02:48:44.220454 15073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:44.167053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:43881 (local address 127.1.210.65:51583)
0806 02:48:44.185126 (+ 18073us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:44.185193 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:44.187665 (+ 2472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:44.189771 (+ 2106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:44.189784 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:44.190487 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:44.191467 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:44.191498 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:44.194286 (+ 2788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:44.194292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:44.199348 (+ 5056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:44.199375 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:44.201871 (+ 2496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:44.201883 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:44.202282 (+ 399us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:44.203216 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:44.203257 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:44.206283 (+ 3026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:44.211785 (+ 5502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:44.211796 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:44.211816 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:44.212518 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:44.213157 (+ 639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:44.213162 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:44.213167 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:44.213449 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:44.214606 (+ 1157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:44.214615 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:44.215119 (+ 504us) client_negotiation.cc:769] Sending connection context
0806 02:48:44.215426 (+ 307us) client_negotiation.cc:240] Negotiation successful
0806 02:48:44.216048 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17760,"thread_start_us":17570,"threads_started":1}
I20260806 02:48:44.224583 15070 heartbeater.cc:347] Connected to a master server at 127.1.210.126:43881
I20260806 02:48:44.224961 15070 heartbeater.cc:464] Registering TS with master...
I20260806 02:48:44.225894 15070 heartbeater.cc:511] Master 127.1.210.126:43881 requested a full tablet report, sending...
W20260806 02:48:44.229540 15070 heartbeater.cc:644] Failed to heartbeat to 127.1.210.126:43881 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 02:48:44.229820 15070 heartbeater.cc:464] Registering TS with master...
I20260806 02:48:44.230173 15070 heartbeater.cc:511] Master 127.1.210.126:43881 requested a full tablet report, sending...
I20260806 02:48:44.231619 15070 heartbeater.cc:464] Registering TS with master...
I20260806 02:48:44.231920 15070 heartbeater.cc:511] Master 127.1.210.126:43881 requested a full tablet report, sending...
W20260806 02:48:44.233143 15070 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:48:44.626246 15074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:44.626644 15074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:44.626850 15074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:44.635640 15074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:44.635797 15074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:44.635989 15074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.66
I20260806 02:48:44.650730 15074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:34445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.210.84:33933/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.66:36883
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.66
--webserver_port=37969
--tserver_master_addrs=127.1.210.126:43881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.66
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:48:44.652562 15074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:44.654435 15074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:44.677314 15088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:44.679767 15089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:44.683041 15074 server_base.cc:1034] running on GCE node
W20260806 02:48:44.687223 15093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:44.688301 15074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:44.692962 15074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:44.694229 15074 hybrid_clock.cc:648] HybridClock initialized: now 1785984524694151 us; error 72 us; skew 500 ppm
I20260806 02:48:44.694664 15074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:48:44 dist-test-slave-8tdl krb5kdc[13853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984524, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:44.699029 15074 init.cc:377] Logged in from keytab as kudu/127.1.210.66@KRBTEST.COM (short username kudu)
I20260806 02:48:44.707775 15074 webserver.cc:460] Webserver started at http://127.1.210.66:37969/ using document root <none> and password file <none>
I20260806 02:48:44.708842 15074 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:44.709029 15074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 02:48:44 dist-test-slave-8tdl krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984524, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for HTTP/127.1.210.84@KRBTEST.COM
I20260806 02:48:44.731444 15074 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:48:44.734611 15101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:44.736395 15074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:48:44.736616 15074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-1/wal
uuid: "4044f79d135f48b2ad8d2253454e625c"
format_stamp: "Formatted at 2026-08-06 02:48:41 on dist-test-slave-8tdl"
server_key: "7d1f684cedefd87f3c9dc89d9bf28904"
server_key_iv: "489073c8238ff52c45f907ca9c7c9c61"
server_key_version: "kuduclusterkey@0"
I20260806 02:48:44.737685 15074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:44.759763 15074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:44.765019 15074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:44.765513 15074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:44.767047 15074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:48:44.768684 15074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:48:44.768814 15074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:44.768955 15074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:48:44.769055 15074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:48:44 dist-test-slave-8tdl krb5kdc[13853](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785984524, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:44.832557 15074 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.66:36883
I20260806 02:48:44.832800 15214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.66:36883 every 8 connection(s)
I20260806 02:48:44.834384 15074 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
I20260806 02:48:44.843182 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 15074
I20260806 02:48:44.843962 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.210.84:33933/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.67:33129
--local_ip_for_outbound_sockets=127.1.210.67
--tserver_master_addrs=127.1.210.126:43881
--webserver_port=36001
--webserver_interface=127.1.210.67
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:34445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.210.84:33933/kms with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:48:44 dist-test-slave-8tdl krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984524, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:48:44.858999 15217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:44.837505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:60197 (local address 127.1.210.126:43881)
0806 02:48:44.837967 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:44.838011 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:44.839337 (+ 1326us) server_negotiation.cc:407] Connection header received
0806 02:48:44.841343 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:44.841349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:44.841448 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:44.841571 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:44.843363 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:44.844377 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:44.846064 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:44.846489 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:44.846630 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:44.850125 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:44.850132 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:44.850137 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:44.850184 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:44.854021 (+ 3837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:44.855146 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:44.855155 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:44.855161 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:44.855335 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:44.855924 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:44.855932 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:44.855939 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:44.856349 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:44.856524 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:44.858167 (+ 1643us) server_negotiation.cc:299] Negotiation successful
0806 02:48:44.858451 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":248,"threads_started":1}
I20260806 02:48:44.861454 15218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:44.838157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:43881 (local address 127.1.210.66:60197)
0806 02:48:44.839161 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:44.839218 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:44.841044 (+ 1826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:44.841830 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:44.841839 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:44.842274 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:44.843031 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:44.843056 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:44.844584 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:44.844588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:44.845886 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:44.845899 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:44.846765 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:44.846774 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:44.847015 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:44.847728 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:44.847751 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:44.849864 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:44.854221 (+ 4357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:44.854232 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:44.854252 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:44.854987 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:44.855492 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:44.855501 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:44.855508 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:44.855776 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:44.856612 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:44.856623 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:44.857124 (+ 501us) client_negotiation.cc:769] Sending connection context
0806 02:48:44.857443 (+ 319us) client_negotiation.cc:240] Negotiation successful
0806 02:48:44.859509 (+ 2066us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":361,"threads_started":1}
I20260806 02:48:44.863389 15215 heartbeater.cc:347] Connected to a master server at 127.1.210.126:43881
I20260806 02:48:44.863770 15215 heartbeater.cc:464] Registering TS with master...
I20260806 02:48:44.864962 15215 heartbeater.cc:511] Master 127.1.210.126:43881 requested a full tablet report, sending...
I20260806 02:48:44.867379 14871 ts_manager.cc:194] Registered new tserver with Master: 4044f79d135f48b2ad8d2253454e625c (127.1.210.66:36883)
I20260806 02:48:44.869683 14871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.66@KRBTEST.COM'} at 127.1.210.66:60197
Aug 06 02:48:45 dist-test-slave-8tdl krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984524, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for HTTP/127.1.210.84@KRBTEST.COM
I20260806 02:48:45.233314 15070 heartbeater.cc:464] Registering TS with master...
I20260806 02:48:45.233820 15070 heartbeater.cc:511] Master 127.1.210.126:43881 requested a full tablet report, sending...
I20260806 02:48:45.235517 14871 ts_manager.cc:194] Registered new tserver with Master: 9aef7f7c5bc7414f8e0df3d807c556db (127.1.210.65:36801)
I20260806 02:48:45.236776 14871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.65@KRBTEST.COM'} at 127.1.210.65:51583
W20260806 02:48:45.361186 15219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:45.361620 15219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:45.361786 15219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:45.373592 15219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:45.373751 15219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:45.373924 15219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.67
I20260806 02:48:45.398602 15219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:34445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.210.84:33933/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.67:33129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.67
--webserver_port=36001
--tserver_master_addrs=127.1.210.126:43881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.67
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:48:45.400393 15219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:45.402086 15219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:45.419817 15225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:45.423457 15226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:45.424444 15228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:45.425665 15219 server_base.cc:1034] running on GCE node
I20260806 02:48:45.426599 15219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:45.428064 15219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:45.429351 15219 hybrid_clock.cc:648] HybridClock initialized: now 1785984525429285 us; error 55 us; skew 500 ppm
I20260806 02:48:45.429775 15219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:48:45 dist-test-slave-8tdl krb5kdc[13853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984525, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:45.434507 15219 init.cc:377] Logged in from keytab as kudu/127.1.210.67@KRBTEST.COM (short username kudu)
I20260806 02:48:45.436671 15219 webserver.cc:460] Webserver started at http://127.1.210.67:36001/ using document root <none> and password file <none>
I20260806 02:48:45.437729 15219 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:45.437897 15219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 02:48:45 dist-test-slave-8tdl krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984525, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for HTTP/127.1.210.84@KRBTEST.COM
I20260806 02:48:45.467649 15219 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260806 02:48:45.471798 15236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:45.473780 15219 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:48:45.473976 15219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-2/wal
uuid: "c0cf376deec64958b6a08c98ee5e67f7"
format_stamp: "Formatted at 2026-08-06 02:48:42 on dist-test-slave-8tdl"
server_key: "71528d723ce60dbb5a3226c3668be73c"
server_key_iv: "6beafdd5b8c8c06d023a06ab65ee52b7"
server_key_version: "kuduclusterkey@0"
I20260806 02:48:45.475400 15219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:45.518514 15219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:45.528358 15219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:45.528764 15219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:45.533140 15219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:48:45.535200 15219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:48:45.535319 15219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:45.535418 15219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:48:45.535485 15219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:48:45 dist-test-slave-8tdl krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984524, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for HTTP/127.1.210.84@KRBTEST.COM
I20260806 02:48:45.872488 15219 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.67:33129
I20260806 02:48:45.872588 15349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.67:33129 every 8 connection(s)
I20260806 02:48:45.874421 15219 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
I20260806 02:48:45.875191 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 15219
I20260806 02:48:45.876037 15215 heartbeater.cc:503] Master 127.1.210.126:43881 was elected leader, sending a full tablet report...
Aug 06 02:48:45 dist-test-slave-8tdl krb5kdc[13853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984525, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:48:45.926406 15352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:45.876744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35568 (local address 127.1.210.126:43881)
0806 02:48:45.888129 (+ 11385us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:45.888137 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:45.890425 (+ 2288us) server_negotiation.cc:407] Connection header received
0806 02:48:45.890514 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:45.890520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:45.890610 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:45.890835 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:45.907394 (+ 16559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:45.908231 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:45.912334 (+ 4103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:45.912793 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:45.912954 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:45.917745 (+ 4791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:45.917752 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:45.917757 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:45.917819 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:45.922719 (+ 4900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:45.923777 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:45.923783 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:45.923787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:45.923967 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:45.924496 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:45.924502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:45.924505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:45.924851 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:45.925098 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:45.925745 (+ 647us) server_negotiation.cc:299] Negotiation successful
0806 02:48:45.926029 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":11229,"thread_start_us":12204,"threads_started":1}
I20260806 02:48:45.927533 15355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:45.890223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:47329 (local address 127.1.210.126:43881)
0806 02:48:45.905946 (+ 15723us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:45.905954 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:45.905977 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:48:45.906059 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:45.906065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:45.906175 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:45.906303 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:45.908844 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:45.910073 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:45.912673 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:45.913426 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:45.913542 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:45.918200 (+ 4658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:45.918212 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:45.918219 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:45.918284 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:45.922485 (+ 4201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:45.923603 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:45.923609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:45.923612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:45.923770 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:45.924358 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:45.924363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:45.924367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:45.924736 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:45.924878 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:45.926928 (+ 2050us) server_negotiation.cc:299] Negotiation successful
0806 02:48:45.927158 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":15557,"thread_start_us":16552,"threads_started":1}
I20260806 02:48:45.928800 15353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:45.884226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:43881 (local address 127.1.210.67:47329)
0806 02:48:45.896850 (+ 12624us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:45.896922 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:45.901349 (+ 4427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:45.906970 (+ 5621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:45.906986 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:45.907684 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:45.908575 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:45.908605 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:45.910302 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:45.910308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:45.912467 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:45.912487 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:45.913870 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:45.913877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:45.914192 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:45.914998 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:45.915044 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:45.917812 (+ 2768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:45.922731 (+ 4919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:45.922743 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:45.922764 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:45.923437 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:45.923904 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:45.923909 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:45.923913 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:45.924220 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:45.924958 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:45.924967 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:45.925493 (+ 526us) client_negotiation.cc:769] Sending connection context
0806 02:48:45.925816 (+ 323us) client_negotiation.cc:240] Negotiation successful
0806 02:48:45.926427 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12213,"thread_start_us":14420,"threads_started":1}
I20260806 02:48:45.930797 15350 heartbeater.cc:347] Connected to a master server at 127.1.210.126:43881
I20260806 02:48:45.931164 15350 heartbeater.cc:464] Registering TS with master...
I20260806 02:48:45.932070 15350 heartbeater.cc:511] Master 127.1.210.126:43881 requested a full tablet report, sending...
I20260806 02:48:45.934046 14870 ts_manager.cc:194] Registered new tserver with Master: c0cf376deec64958b6a08c98ee5e67f7 (127.1.210.67:33129)
I20260806 02:48:45.935453 14870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.67@KRBTEST.COM'} at 127.1.210.67:47329
I20260806 02:48:45.943813 1865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:48:45.961922 15355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:45.947286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35580 (local address 127.1.210.126:43881)
0806 02:48:45.947513 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:45.947520 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:45.947787 (+ 267us) server_negotiation.cc:407] Connection header received
0806 02:48:45.948009 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:45.948014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:45.948093 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:45.948202 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:45.949874 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:45.950626 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:45.952112 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:45.952565 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:45.952709 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:45.955559 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:45.955564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:45.955568 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:45.955606 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:45.958838 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:45.959749 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:45.959755 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:45.959757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:45.959902 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:45.960299 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:45.960352 (+ 53us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:45.960355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:45.960672 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:45.960994 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:45.961312 (+ 318us) server_negotiation.cc:299] Negotiation successful
0806 02:48:45.961562 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 02:48:45.967156 14870 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:48:45.974879 14870 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:48:46.004042 15374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:45.990306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50252 (local address 127.1.210.67:33129)
0806 02:48:45.995551 (+ 5245us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:45.995559 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:45.995582 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:48:45.995674 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:45.995680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:45.995881 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:45.996074 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:45.997374 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:45.998998 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:46.001646 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:46.002650 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:46.002840 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:46.003125 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:46.003459 (+ 334us) server_negotiation.cc:299] Negotiation successful
0806 02:48:46.003651 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5083,"thread_start_us":4945,"threads_started":1}
I20260806 02:48:46.004266 15372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:45.989838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:33129 (local address 127.0.0.1:50252)
0806 02:48:45.990920 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:45.990973 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:45.991215 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:45.996056 (+ 4841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:45.996064 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:45.996094 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:45.996687 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:45.996705 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:45.999144 (+ 2439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:45.999149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:46.001479 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:46.001492 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:46.002914 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:46.002922 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:46.003145 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:46.003222 (+ 77us) client_negotiation.cc:769] Sending connection context
0806 02:48:46.003384 (+ 162us) client_negotiation.cc:240] Negotiation successful
0806 02:48:46.003621 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"mutex_wait_us":113,"thread_start_us":1182,"threads_started":1}
I20260806 02:48:46.004767 15376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:45.992127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:36801 (local address 127.0.0.1:48132)
0806 02:48:45.992994 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:45.993045 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:45.993240 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:45.995745 (+ 2505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:45.995752 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:45.995789 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:45.996687 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:45.996705 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:45.998932 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:45.998937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:46.001564 (+ 2627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:46.001578 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:46.003899 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:46.003904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:46.004093 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:46.004127 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 02:48:46.004200 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 02:48:46.004348 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"mutex_wait_us":111,"thread_start_us":885,"threads_started":1}
I20260806 02:48:46.005352 15373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:45.990201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:36883 (local address 127.0.0.1:41072)
0806 02:48:45.991992 (+ 1791us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:45.992015 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:45.992160 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:45.995937 (+ 3777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:45.995942 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:45.996010 (+ 68us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:45.996853 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:45.996870 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:45.998416 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:45.998424 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:46.001480 (+ 3056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:46.001498 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:46.003779 (+ 2281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:46.003788 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:46.004011 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:46.004076 (+ 65us) client_negotiation.cc:769] Sending connection context
0806 02:48:46.004168 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 02:48:46.004698 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1635,"mutex_wait_us":257,"spinlock_wait_cycles":1408,"thread_start_us":1746,"threads_started":1}
I20260806 02:48:46.005775 15377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:45.992765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48132 (local address 127.1.210.65:36801)
0806 02:48:45.994985 (+ 2220us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:45.994999 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:45.995040 (+ 41us) server_negotiation.cc:407] Connection header received
0806 02:48:45.995174 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:45.995187 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:45.995513 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:45.995726 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:45.997019 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:45.998768 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:46.001740 (+ 2972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:46.003345 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:46.003496 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:46.003747 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:46.005174 (+ 1427us) server_negotiation.cc:299] Negotiation successful
0806 02:48:46.005390 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1976,"thread_start_us":1664,"threads_started":1}
I20260806 02:48:46.006927 15375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:45.990643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41072 (local address 127.1.210.66:36883)
0806 02:48:45.995265 (+ 4622us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:45.995275 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:45.995311 (+ 36us) server_negotiation.cc:407] Connection header received
0806 02:48:45.995424 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:45.995433 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:45.995681 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:45.996090 (+ 409us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:45.997065 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:45.998275 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:46.001685 (+ 3410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:46.003344 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:46.003484 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:46.006297 (+ 2813us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:46.006370 (+ 73us) server_negotiation.cc:299] Negotiation successful
0806 02:48:46.006545 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4434,"thread_start_us":4165,"threads_started":1}
I20260806 02:48:46.009596 15004 tablet_service.cc:1467] Processing CreateTablet for tablet 9d65c8bb842f40b8b7b73b408d8a4f9f (DEFAULT_TABLE table=test-table [id=ce86b31e69f64bbd928ca5ac95274f42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:48:46.009370 15285 tablet_service.cc:1467] Processing CreateTablet for tablet 9d65c8bb842f40b8b7b73b408d8a4f9f (DEFAULT_TABLE table=test-table [id=ce86b31e69f64bbd928ca5ac95274f42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:48:46.009958 15149 tablet_service.cc:1467] Processing CreateTablet for tablet 9d65c8bb842f40b8b7b73b408d8a4f9f (DEFAULT_TABLE table=test-table [id=ce86b31e69f64bbd928ca5ac95274f42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:48:46.011129 15285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d65c8bb842f40b8b7b73b408d8a4f9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:48:46.011148 15004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d65c8bb842f40b8b7b73b408d8a4f9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:48:46.011459 15149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d65c8bb842f40b8b7b73b408d8a4f9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:48:46.026696 15378 tablet_bootstrap.cc:492] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 4044f79d135f48b2ad8d2253454e625c: Bootstrap starting.
I20260806 02:48:46.030162 15378 tablet_bootstrap.cc:654] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 4044f79d135f48b2ad8d2253454e625c: Neither blocks nor log segments found. Creating new log.
I20260806 02:48:46.031374 15378 log.cc:824] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 4044f79d135f48b2ad8d2253454e625c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:46.032917 15379 tablet_bootstrap.cc:492] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 9aef7f7c5bc7414f8e0df3d807c556db: Bootstrap starting.
I20260806 02:48:46.035097 15378 tablet_bootstrap.cc:492] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 4044f79d135f48b2ad8d2253454e625c: No bootstrap required, opened a new log
I20260806 02:48:46.035400 15378 ts_tablet_manager.cc:1397] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 4044f79d135f48b2ad8d2253454e625c: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260806 02:48:46.035409 15380 tablet_bootstrap.cc:492] T 9d65c8bb842f40b8b7b73b408d8a4f9f P c0cf376deec64958b6a08c98ee5e67f7: Bootstrap starting.
I20260806 02:48:46.039378 15380 tablet_bootstrap.cc:654] T 9d65c8bb842f40b8b7b73b408d8a4f9f P c0cf376deec64958b6a08c98ee5e67f7: Neither blocks nor log segments found. Creating new log.
I20260806 02:48:46.040747 15380 log.cc:824] T 9d65c8bb842f40b8b7b73b408d8a4f9f P c0cf376deec64958b6a08c98ee5e67f7: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:46.041708 15379 tablet_bootstrap.cc:654] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 9aef7f7c5bc7414f8e0df3d807c556db: Neither blocks nor log segments found. Creating new log.
I20260806 02:48:46.043151 15379 log.cc:824] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 9aef7f7c5bc7414f8e0df3d807c556db: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:46.044658 15380 tablet_bootstrap.cc:492] T 9d65c8bb842f40b8b7b73b408d8a4f9f P c0cf376deec64958b6a08c98ee5e67f7: No bootstrap required, opened a new log
I20260806 02:48:46.044952 15380 ts_tablet_manager.cc:1397] T 9d65c8bb842f40b8b7b73b408d8a4f9f P c0cf376deec64958b6a08c98ee5e67f7: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260806 02:48:46.046515 15378 raft_consensus.cc:348] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 4044f79d135f48b2ad8d2253454e625c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cf376deec64958b6a08c98ee5e67f7" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 33129 } } peers { permanent_uuid: "4044f79d135f48b2ad8d2253454e625c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 36883 } } peers { permanent_uuid: "9aef7f7c5bc7414f8e0df3d807c556db" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 36801 } }
I20260806 02:48:46.047040 15378 raft_consensus.cc:374] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 4044f79d135f48b2ad8d2253454e625c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:48:46.047158 15378 raft_consensus.cc:729] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 4044f79d135f48b2ad8d2253454e625c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4044f79d135f48b2ad8d2253454e625c, State: Initialized, Role: FOLLOWER
I20260806 02:48:46.047299 15379 tablet_bootstrap.cc:492] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 9aef7f7c5bc7414f8e0df3d807c556db: No bootstrap required, opened a new log
I20260806 02:48:46.047626 15379 ts_tablet_manager.cc:1397] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 9aef7f7c5bc7414f8e0df3d807c556db: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.000s
I20260806 02:48:46.047827 15378 consensus_queue.cc:260] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 4044f79d135f48b2ad8d2253454e625c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cf376deec64958b6a08c98ee5e67f7" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 33129 } } peers { permanent_uuid: "4044f79d135f48b2ad8d2253454e625c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 36883 } } peers { permanent_uuid: "9aef7f7c5bc7414f8e0df3d807c556db" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 36801 } }
I20260806 02:48:46.049726 15378 ts_tablet_manager.cc:1428] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 4044f79d135f48b2ad8d2253454e625c: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260806 02:48:46.055154 15380 raft_consensus.cc:348] T 9d65c8bb842f40b8b7b73b408d8a4f9f P c0cf376deec64958b6a08c98ee5e67f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cf376deec64958b6a08c98ee5e67f7" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 33129 } } peers { permanent_uuid: "4044f79d135f48b2ad8d2253454e625c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 36883 } } peers { permanent_uuid: "9aef7f7c5bc7414f8e0df3d807c556db" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 36801 } }
I20260806 02:48:46.055636 15380 raft_consensus.cc:374] T 9d65c8bb842f40b8b7b73b408d8a4f9f P c0cf376deec64958b6a08c98ee5e67f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:48:46.055769 15380 raft_consensus.cc:729] T 9d65c8bb842f40b8b7b73b408d8a4f9f P c0cf376deec64958b6a08c98ee5e67f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0cf376deec64958b6a08c98ee5e67f7, State: Initialized, Role: FOLLOWER
I20260806 02:48:46.056555 15380 consensus_queue.cc:260] T 9d65c8bb842f40b8b7b73b408d8a4f9f P c0cf376deec64958b6a08c98ee5e67f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cf376deec64958b6a08c98ee5e67f7" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 33129 } } peers { permanent_uuid: "4044f79d135f48b2ad8d2253454e625c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 36883 } } peers { permanent_uuid: "9aef7f7c5bc7414f8e0df3d807c556db" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 36801 } }
I20260806 02:48:46.057914 15350 heartbeater.cc:503] Master 127.1.210.126:43881 was elected leader, sending a full tablet report...
I20260806 02:48:46.058436 15380 ts_tablet_manager.cc:1428] T 9d65c8bb842f40b8b7b73b408d8a4f9f P c0cf376deec64958b6a08c98ee5e67f7: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260806 02:48:46.058439 15379 raft_consensus.cc:348] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 9aef7f7c5bc7414f8e0df3d807c556db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cf376deec64958b6a08c98ee5e67f7" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 33129 } } peers { permanent_uuid: "4044f79d135f48b2ad8d2253454e625c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 36883 } } peers { permanent_uuid: "9aef7f7c5bc7414f8e0df3d807c556db" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 36801 } }
I20260806 02:48:46.058951 15379 raft_consensus.cc:374] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 9aef7f7c5bc7414f8e0df3d807c556db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:48:46.059090 15379 raft_consensus.cc:729] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 9aef7f7c5bc7414f8e0df3d807c556db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aef7f7c5bc7414f8e0df3d807c556db, State: Initialized, Role: FOLLOWER
I20260806 02:48:46.059864 15379 consensus_queue.cc:260] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 9aef7f7c5bc7414f8e0df3d807c556db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cf376deec64958b6a08c98ee5e67f7" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 33129 } } peers { permanent_uuid: "4044f79d135f48b2ad8d2253454e625c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 36883 } } peers { permanent_uuid: "9aef7f7c5bc7414f8e0df3d807c556db" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 36801 } }
I20260806 02:48:46.061136 15070 heartbeater.cc:503] Master 127.1.210.126:43881 was elected leader, sending a full tablet report...
I20260806 02:48:46.061893 15379 ts_tablet_manager.cc:1428] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 9aef7f7c5bc7414f8e0df3d807c556db: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
W20260806 02:48:46.087078 15216 tablet.cc:2369] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 4044f79d135f48b2ad8d2253454e625c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:48:46.151124 15354 tablet.cc:2369] T 9d65c8bb842f40b8b7b73b408d8a4f9f P c0cf376deec64958b6a08c98ee5e67f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:48:46.160032 15386 raft_consensus.cc:482] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 9aef7f7c5bc7414f8e0df3d807c556db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:48:46.160485 15386 raft_consensus.cc:504] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 9aef7f7c5bc7414f8e0df3d807c556db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cf376deec64958b6a08c98ee5e67f7" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 33129 } } peers { permanent_uuid: "4044f79d135f48b2ad8d2253454e625c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 36883 } } peers { permanent_uuid: "9aef7f7c5bc7414f8e0df3d807c556db" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 36801 } }
I20260806 02:48:46.162741 15386 leader_election.cc:288] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 9aef7f7c5bc7414f8e0df3d807c556db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0cf376deec64958b6a08c98ee5e67f7 (127.1.210.67:33129), 4044f79d135f48b2ad8d2253454e625c (127.1.210.66:36883)
I20260806 02:48:46.163697 15384 raft_consensus.cc:482] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 4044f79d135f48b2ad8d2253454e625c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:48:46.164018 15384 raft_consensus.cc:504] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 4044f79d135f48b2ad8d2253454e625c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cf376deec64958b6a08c98ee5e67f7" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 33129 } } peers { permanent_uuid: "4044f79d135f48b2ad8d2253454e625c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 36883 } } peers { permanent_uuid: "9aef7f7c5bc7414f8e0df3d807c556db" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 36801 } }
I20260806 02:48:46.166092 15384 leader_election.cc:288] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 4044f79d135f48b2ad8d2253454e625c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0cf376deec64958b6a08c98ee5e67f7 (127.1.210.67:33129), 9aef7f7c5bc7414f8e0df3d807c556db (127.1.210.65:36801)
I20260806 02:48:46.173038 15374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:46.163764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:45385 (local address 127.1.210.67:33129)
0806 02:48:46.163930 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:46.163935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:46.164744 (+ 809us) server_negotiation.cc:407] Connection header received
0806 02:48:46.165055 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:46.165062 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:46.165130 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:46.165181 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:46.166076 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:46.167264 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:46.170169 (+ 2905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:46.171490 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:46.171644 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:46.171729 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:46.172495 (+ 766us) server_negotiation.cc:299] Negotiation successful
0806 02:48:46.172645 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 02:48:46.173692 15375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:46.163205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:39703 (local address 127.1.210.66:36883)
0806 02:48:46.163458 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:46.163467 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:46.164923 (+ 1456us) server_negotiation.cc:407] Connection header received
0806 02:48:46.165053 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:46.165059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:46.165148 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:46.165268 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:46.166106 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:46.167293 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:46.170508 (+ 3215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:46.171862 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:46.171990 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:46.172159 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:46.172808 (+ 649us) server_negotiation.cc:299] Negotiation successful
0806 02:48:46.173078 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 02:48:46.174715 15387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:46.163198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:36883 (local address 127.1.210.65:39703)
0806 02:48:46.164805 (+ 1607us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:46.164819 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:46.164930 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:46.165299 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:46.165306 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:46.165344 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:46.165890 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:46.165902 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:46.167478 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:46.167484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:46.170319 (+ 2835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:46.170333 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:46.172409 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:46.172418 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:46.172627 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:46.172657 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 02:48:46.174037 (+ 1380us) client_negotiation.cc:240] Negotiation successful
0806 02:48:46.174210 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1497,"mutex_wait_us":3,"spinlock_wait_cycles":1280,"thread_start_us":906,"threads_started":1}
I20260806 02:48:46.175840 15388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:46.163097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:33129 (local address 127.1.210.65:45385)
0806 02:48:46.164591 (+ 1494us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:46.164615 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:46.164874 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:46.165385 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:46.165392 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:46.165445 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:46.165897 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:46.165905 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:46.167501 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:46.167508 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:46.169957 (+ 2449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:46.169975 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:46.171946 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:46.171953 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:46.172265 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:46.172310 (+ 45us) client_negotiation.cc:769] Sending connection context
0806 02:48:46.175270 (+ 2960us) client_negotiation.cc:240] Negotiation successful
0806 02:48:46.175414 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1232,"mutex_wait_us":26,"spinlock_wait_cycles":38144,"thread_start_us":1396,"threads_started":1}
I20260806 02:48:46.177261 15305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d65c8bb842f40b8b7b73b408d8a4f9f" candidate_uuid: "9aef7f7c5bc7414f8e0df3d807c556db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0cf376deec64958b6a08c98ee5e67f7" is_pre_election: true
I20260806 02:48:46.177815 15305 raft_consensus.cc:2454] T 9d65c8bb842f40b8b7b73b408d8a4f9f P c0cf376deec64958b6a08c98ee5e67f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9aef7f7c5bc7414f8e0df3d807c556db in term 0.
I20260806 02:48:46.178581 14959 leader_election.cc:302] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 9aef7f7c5bc7414f8e0df3d807c556db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9aef7f7c5bc7414f8e0df3d807c556db, c0cf376deec64958b6a08c98ee5e67f7; no voters:
I20260806 02:48:46.179081 15386 raft_consensus.cc:2781] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 9aef7f7c5bc7414f8e0df3d807c556db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:48:46.179207 15386 raft_consensus.cc:482] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 9aef7f7c5bc7414f8e0df3d807c556db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:48:46.179296 15386 raft_consensus.cc:3037] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 9aef7f7c5bc7414f8e0df3d807c556db [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:48:46.180944 15374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:46.166657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:35373 (local address 127.1.210.67:33129)
0806 02:48:46.173354 (+ 6697us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:46.173360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:46.173379 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:48:46.173450 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:46.173455 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:46.173549 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:46.173656 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:46.174499 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:46.175632 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:46.178349 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:46.179616 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:46.179772 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:46.179859 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:46.180506 (+ 647us) server_negotiation.cc:299] Negotiation successful
0806 02:48:46.180649 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6592,"thread_start_us":9607,"threads_started":1}
I20260806 02:48:46.181777 15390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:46.166282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:33129 (local address 127.1.210.66:35373)
0806 02:48:46.168022 (+ 1740us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:46.168041 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:46.168183 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:46.173834 (+ 5651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:46.173839 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:46.173869 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:46.174323 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:46.174339 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:46.175796 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:46.175802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:46.178133 (+ 2331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:46.178149 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:46.180077 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:46.180085 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:46.180302 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:46.180342 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 02:48:46.181156 (+ 814us) client_negotiation.cc:240] Negotiation successful
0806 02:48:46.181327 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1606,"mutex_wait_us":3,"thread_start_us":14454,"threads_started":1}
I20260806 02:48:46.181986 15169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d65c8bb842f40b8b7b73b408d8a4f9f" candidate_uuid: "9aef7f7c5bc7414f8e0df3d807c556db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4044f79d135f48b2ad8d2253454e625c" is_pre_election: true
I20260806 02:48:46.182504 15169 raft_consensus.cc:2454] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 4044f79d135f48b2ad8d2253454e625c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9aef7f7c5bc7414f8e0df3d807c556db in term 0.
I20260806 02:48:46.182755 15386 raft_consensus.cc:504] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 9aef7f7c5bc7414f8e0df3d807c556db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cf376deec64958b6a08c98ee5e67f7" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 33129 } } peers { permanent_uuid: "4044f79d135f48b2ad8d2253454e625c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 36883 } } peers { permanent_uuid: "9aef7f7c5bc7414f8e0df3d807c556db" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 36801 } }
I20260806 02:48:46.183130 15305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d65c8bb842f40b8b7b73b408d8a4f9f" candidate_uuid: "4044f79d135f48b2ad8d2253454e625c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0cf376deec64958b6a08c98ee5e67f7" is_pre_election: true
I20260806 02:48:46.183357 15305 raft_consensus.cc:2454] T 9d65c8bb842f40b8b7b73b408d8a4f9f P c0cf376deec64958b6a08c98ee5e67f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4044f79d135f48b2ad8d2253454e625c in term 0.
I20260806 02:48:46.183947 15386 leader_election.cc:288] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 9aef7f7c5bc7414f8e0df3d807c556db [CANDIDATE]: Term 1 election: Requested vote from peers c0cf376deec64958b6a08c98ee5e67f7 (127.1.210.67:33129), 4044f79d135f48b2ad8d2253454e625c (127.1.210.66:36883)
I20260806 02:48:46.184018 15305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d65c8bb842f40b8b7b73b408d8a4f9f" candidate_uuid: "9aef7f7c5bc7414f8e0df3d807c556db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0cf376deec64958b6a08c98ee5e67f7"
I20260806 02:48:46.184233 15305 raft_consensus.cc:3037] T 9d65c8bb842f40b8b7b73b408d8a4f9f P c0cf376deec64958b6a08c98ee5e67f7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:48:46.184230 15104 leader_election.cc:302] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 4044f79d135f48b2ad8d2253454e625c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4044f79d135f48b2ad8d2253454e625c, c0cf376deec64958b6a08c98ee5e67f7; no voters:
I20260806 02:48:46.184660 15384 raft_consensus.cc:2781] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 4044f79d135f48b2ad8d2253454e625c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:48:46.184821 15384 raft_consensus.cc:482] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 4044f79d135f48b2ad8d2253454e625c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:48:46.184926 15384 raft_consensus.cc:3037] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 4044f79d135f48b2ad8d2253454e625c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:48:46.185251 15377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:46.167123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:53817 (local address 127.1.210.65:36801)
0806 02:48:46.167339 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:46.167347 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:46.177138 (+ 9791us) server_negotiation.cc:407] Connection header received
0806 02:48:46.177370 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:46.177376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:46.177445 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:46.177519 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:46.178505 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:46.179672 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:46.182784 (+ 3112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:46.184207 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:46.184369 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:46.184442 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:46.184868 (+ 426us) server_negotiation.cc:299] Negotiation successful
0806 02:48:46.185002 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260806 02:48:46.185263 15071 tablet.cc:2369] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 9aef7f7c5bc7414f8e0df3d807c556db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:48:46.186358 15389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:46.166657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:36801 (local address 127.1.210.66:53817)
0806 02:48:46.177036 (+ 10379us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:46.177058 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:46.177233 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:46.177785 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:46.177790 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:46.177815 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:46.178320 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:46.178332 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:46.179971 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:46.179978 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:46.182548 (+ 2570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:46.182567 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:46.184445 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:46.184452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:46.184620 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:46.184666 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 02:48:46.184771 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 02:48:46.184954 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10236,"thread_start_us":1201,"threads_started":1}
I20260806 02:48:46.187453 15169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d65c8bb842f40b8b7b73b408d8a4f9f" candidate_uuid: "9aef7f7c5bc7414f8e0df3d807c556db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4044f79d135f48b2ad8d2253454e625c"
I20260806 02:48:46.187636 15024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d65c8bb842f40b8b7b73b408d8a4f9f" candidate_uuid: "4044f79d135f48b2ad8d2253454e625c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9aef7f7c5bc7414f8e0df3d807c556db" is_pre_election: true
I20260806 02:48:46.188174 15024 raft_consensus.cc:2379] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 9aef7f7c5bc7414f8e0df3d807c556db [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4044f79d135f48b2ad8d2253454e625c in current term 1: Already voted for candidate 9aef7f7c5bc7414f8e0df3d807c556db in this term.
I20260806 02:48:46.188071 15384 raft_consensus.cc:504] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 4044f79d135f48b2ad8d2253454e625c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cf376deec64958b6a08c98ee5e67f7" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 33129 } } peers { permanent_uuid: "4044f79d135f48b2ad8d2253454e625c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 36883 } } peers { permanent_uuid: "9aef7f7c5bc7414f8e0df3d807c556db" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 36801 } }
I20260806 02:48:46.188432 15305 raft_consensus.cc:2454] T 9d65c8bb842f40b8b7b73b408d8a4f9f P c0cf376deec64958b6a08c98ee5e67f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9aef7f7c5bc7414f8e0df3d807c556db in term 1.
I20260806 02:48:46.188813 15384 leader_election.cc:288] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 4044f79d135f48b2ad8d2253454e625c [CANDIDATE]: Term 1 election: Requested vote from peers c0cf376deec64958b6a08c98ee5e67f7 (127.1.210.67:33129), 9aef7f7c5bc7414f8e0df3d807c556db (127.1.210.65:36801)
I20260806 02:48:46.188992 14959 leader_election.cc:302] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 9aef7f7c5bc7414f8e0df3d807c556db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9aef7f7c5bc7414f8e0df3d807c556db, c0cf376deec64958b6a08c98ee5e67f7; no voters:
I20260806 02:48:46.189204 15169 raft_consensus.cc:2379] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 4044f79d135f48b2ad8d2253454e625c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9aef7f7c5bc7414f8e0df3d807c556db in current term 1: Already voted for candidate 4044f79d135f48b2ad8d2253454e625c in this term.
I20260806 02:48:46.189363 15386 raft_consensus.cc:2781] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 9aef7f7c5bc7414f8e0df3d807c556db [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:48:46.189936 15386 raft_consensus.cc:686] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 9aef7f7c5bc7414f8e0df3d807c556db [term 1 LEADER]: Becoming Leader. State: Replica: 9aef7f7c5bc7414f8e0df3d807c556db, State: Running, Role: LEADER
I20260806 02:48:46.189944 15024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d65c8bb842f40b8b7b73b408d8a4f9f" candidate_uuid: "4044f79d135f48b2ad8d2253454e625c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9aef7f7c5bc7414f8e0df3d807c556db"
I20260806 02:48:46.190516 15386 consensus_queue.cc:237] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 9aef7f7c5bc7414f8e0df3d807c556db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cf376deec64958b6a08c98ee5e67f7" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 33129 } } peers { permanent_uuid: "4044f79d135f48b2ad8d2253454e625c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 36883 } } peers { permanent_uuid: "9aef7f7c5bc7414f8e0df3d807c556db" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 36801 } }
I20260806 02:48:46.191334 15305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d65c8bb842f40b8b7b73b408d8a4f9f" candidate_uuid: "4044f79d135f48b2ad8d2253454e625c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0cf376deec64958b6a08c98ee5e67f7"
I20260806 02:48:46.191700 15305 raft_consensus.cc:2379] T 9d65c8bb842f40b8b7b73b408d8a4f9f P c0cf376deec64958b6a08c98ee5e67f7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4044f79d135f48b2ad8d2253454e625c in current term 1: Already voted for candidate 9aef7f7c5bc7414f8e0df3d807c556db in this term.
I20260806 02:48:46.192363 15104 leader_election.cc:302] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 4044f79d135f48b2ad8d2253454e625c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4044f79d135f48b2ad8d2253454e625c; no voters: 9aef7f7c5bc7414f8e0df3d807c556db, c0cf376deec64958b6a08c98ee5e67f7
I20260806 02:48:46.192713 15384 raft_consensus.cc:2726] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 4044f79d135f48b2ad8d2253454e625c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 02:48:46.196049 14871 catalog_manager.cc:5495] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 9aef7f7c5bc7414f8e0df3d807c556db reported cstate change: term changed from 0 to 1, leader changed from <none> to 9aef7f7c5bc7414f8e0df3d807c556db (127.1.210.65). New cstate: current_term: 1 leader_uuid: "9aef7f7c5bc7414f8e0df3d807c556db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0cf376deec64958b6a08c98ee5e67f7" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 33129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4044f79d135f48b2ad8d2253454e625c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 36883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9aef7f7c5bc7414f8e0df3d807c556db" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 36801 } health_report { overall_health: HEALTHY } } }
I20260806 02:48:46.249590 15377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:46.243552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48148 (local address 127.1.210.65:36801)
0806 02:48:46.243810 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:46.243817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:46.243958 (+ 141us) server_negotiation.cc:407] Connection header received
0806 02:48:46.244272 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:46.244276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:46.244358 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:46.244522 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:48:46.244941 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:46.245724 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:46.247299 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:46.247731 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:46.247865 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:46.248396 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:48:46.248932 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:48:46.249112 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:46.249197 (+ 85us) server_negotiation.cc:299] Negotiation successful
0806 02:48:46.249290 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 02:48:46.258476 15305 raft_consensus.cc:1261] T 9d65c8bb842f40b8b7b73b408d8a4f9f P c0cf376deec64958b6a08c98ee5e67f7 [term 1 FOLLOWER]: Refusing update from remote peer 9aef7f7c5bc7414f8e0df3d807c556db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:48:46.258569 15169 raft_consensus.cc:1261] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 4044f79d135f48b2ad8d2253454e625c [term 1 FOLLOWER]: Refusing update from remote peer 9aef7f7c5bc7414f8e0df3d807c556db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:48:46.259462 15386 consensus_queue.cc:1035] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 9aef7f7c5bc7414f8e0df3d807c556db [LEADER]: Connected to new peer: Peer: permanent_uuid: "4044f79d135f48b2ad8d2253454e625c" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 36883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:48:46.259789 15392 consensus_queue.cc:1035] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 9aef7f7c5bc7414f8e0df3d807c556db [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0cf376deec64958b6a08c98ee5e67f7" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 33129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:48:46.289218 14871 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:48:46.289525 14871 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:48:46.293597 14871 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 491706754f864fd3a7bddd91f4ebaf83: Sending DeleteTablet for 3 replicas of tablet 9d65c8bb842f40b8b7b73b408d8a4f9f
I20260806 02:48:46.295065 15004 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d65c8bb842f40b8b7b73b408d8a4f9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:48:46 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:48132
I20260806 02:48:46.295059 15285 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d65c8bb842f40b8b7b73b408d8a4f9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:48:46 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:50252
I20260806 02:48:46.296052 15149 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d65c8bb842f40b8b7b73b408d8a4f9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:48:46 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:41072
I20260806 02:48:46.296468 15405 tablet_replica.cc:330] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 9aef7f7c5bc7414f8e0df3d807c556db: stopping tablet replica
I20260806 02:48:46.296943 15405 raft_consensus.cc:2229] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 9aef7f7c5bc7414f8e0df3d807c556db [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:48:46.297765 15405 raft_consensus.cc:2258] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 9aef7f7c5bc7414f8e0df3d807c556db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:48:46.298832 15406 tablet_replica.cc:330] T 9d65c8bb842f40b8b7b73b408d8a4f9f P c0cf376deec64958b6a08c98ee5e67f7: stopping tablet replica
I20260806 02:48:46.299335 15406 raft_consensus.cc:2229] T 9d65c8bb842f40b8b7b73b408d8a4f9f P c0cf376deec64958b6a08c98ee5e67f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:48:46.299917 15406 raft_consensus.cc:2258] T 9d65c8bb842f40b8b7b73b408d8a4f9f P c0cf376deec64958b6a08c98ee5e67f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:48:46.300913 15405 ts_tablet_manager.cc:1905] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 9aef7f7c5bc7414f8e0df3d807c556db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:48:46.303040 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 14908
I20260806 02:48:46.304137 15407 tablet_replica.cc:330] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 4044f79d135f48b2ad8d2253454e625c: stopping tablet replica
I20260806 02:48:46.304456 15406 ts_tablet_manager.cc:1905] T 9d65c8bb842f40b8b7b73b408d8a4f9f P c0cf376deec64958b6a08c98ee5e67f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:48:46.304571 15407 raft_consensus.cc:2229] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 4044f79d135f48b2ad8d2253454e625c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:48:46.305052 15407 raft_consensus.cc:2258] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 4044f79d135f48b2ad8d2253454e625c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:48:46.307241 15407 ts_tablet_manager.cc:1905] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 4044f79d135f48b2ad8d2253454e625c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:48:46.311929 15406 ts_tablet_manager.cc:1918] T 9d65c8bb842f40b8b7b73b408d8a4f9f P c0cf376deec64958b6a08c98ee5e67f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:48:46.312099 15406 log.cc:1196] T 9d65c8bb842f40b8b7b73b408d8a4f9f P c0cf376deec64958b6a08c98ee5e67f7: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-2/wal/wals/9d65c8bb842f40b8b7b73b408d8a4f9f
I20260806 02:48:46.312656 15406 ts_tablet_manager.cc:1939] T 9d65c8bb842f40b8b7b73b408d8a4f9f P c0cf376deec64958b6a08c98ee5e67f7: Deleting consensus metadata
I20260806 02:48:46.314234 15407 ts_tablet_manager.cc:1918] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 4044f79d135f48b2ad8d2253454e625c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:48:46.314400 15407 log.cc:1196] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 4044f79d135f48b2ad8d2253454e625c: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-1/wal/wals/9d65c8bb842f40b8b7b73b408d8a4f9f
I20260806 02:48:46.314697 14857 catalog_manager.cc:4841] TS c0cf376deec64958b6a08c98ee5e67f7 (127.1.210.67:33129): tablet 9d65c8bb842f40b8b7b73b408d8a4f9f (table test-table [id=ce86b31e69f64bbd928ca5ac95274f42]) successfully deleted
I20260806 02:48:46.315030 15407 ts_tablet_manager.cc:1939] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 4044f79d135f48b2ad8d2253454e625c: Deleting consensus metadata
I20260806 02:48:46.316672 15405 ts_tablet_manager.cc:1918] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 9aef7f7c5bc7414f8e0df3d807c556db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:48:46.316854 14856 catalog_manager.cc:4841] TS 4044f79d135f48b2ad8d2253454e625c (127.1.210.66:36883): tablet 9d65c8bb842f40b8b7b73b408d8a4f9f (table test-table [id=ce86b31e69f64bbd928ca5ac95274f42]) successfully deleted
I20260806 02:48:46.316944 15405 log.cc:1196] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 9aef7f7c5bc7414f8e0df3d807c556db: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984304136196-1865-0/minicluster-data/ts-0/wal/wals/9d65c8bb842f40b8b7b73b408d8a4f9f
I20260806 02:48:46.317698 15405 ts_tablet_manager.cc:1939] T 9d65c8bb842f40b8b7b73b408d8a4f9f P 9aef7f7c5bc7414f8e0df3d807c556db: Deleting consensus metadata
I20260806 02:48:46.319866 14855 catalog_manager.cc:4841] TS 9aef7f7c5bc7414f8e0df3d807c556db (127.1.210.65:36801): tablet 9d65c8bb842f40b8b7b73b408d8a4f9f (table test-table [id=ce86b31e69f64bbd928ca5ac95274f42]) successfully deleted
I20260806 02:48:46.325438 15377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:46.306169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48150 (local address 127.1.210.65:36801)
0806 02:48:46.306834 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:46.306841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:46.306860 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:48:46.306921 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:46.306926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:46.307012 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:46.307110 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:46.310074 (+ 2964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:46.311243 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:46.313373 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:46.314322 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:46.314447 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:46.318535 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:46.318548 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:46.318562 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:46.318603 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:46.322295 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:46.323175 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:46.323184 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:46.323186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:46.323317 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:46.323741 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:46.323747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:46.323757 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:46.324205 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:46.324547 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:46.324865 (+ 318us) server_negotiation.cc:299] Negotiation successful
0806 02:48:46.325108 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483}
I20260806 02:48:46.326218 15064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:48:46.429141 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 14908
I20260806 02:48:46.448945 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 15074
I20260806 02:48:46.463900 15375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:46.450139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41076 (local address 127.1.210.66:36883)
0806 02:48:46.450333 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:46.450339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:46.450359 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:48:46.450484 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:46.450488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:46.450563 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:46.450651 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:46.452149 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:46.452953 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:46.454331 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:46.455047 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:46.455156 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:46.457795 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:46.457810 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:46.457826 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:46.457863 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:46.460964 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:46.461717 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:46.461724 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:46.461726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:46.461860 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:46.462282 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:46.462287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:46.462290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:46.462805 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:46.463116 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:46.463456 (+ 340us) server_negotiation.cc:299] Negotiation successful
0806 02:48:46.463629 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 02:48:46.464655 15210 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:48:46.562319 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 15074
I20260806 02:48:46.580385 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 15219
I20260806 02:48:46.594929 15374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:46.581346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50258 (local address 127.1.210.67:33129)
0806 02:48:46.581550 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:46.581555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:46.581687 (+ 132us) server_negotiation.cc:407] Connection header received
0806 02:48:46.581782 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:46.581787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:46.581941 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:46.582039 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:46.583567 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:46.584309 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:46.585673 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:46.586088 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:46.586187 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:46.589031 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:46.589048 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:46.589063 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:46.589102 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:46.591955 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:46.592706 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:46.592713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:46.592716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:46.592853 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:46.593209 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:46.593213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:46.593214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:46.593611 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:46.593859 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:46.594135 (+ 276us) server_negotiation.cc:299] Negotiation successful
0806 02:48:46.594335 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 02:48:46.595618 15345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:48:46.688421 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 15219
I20260806 02:48:46.705570 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 14838
I20260806 02:48:46.707113 14901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:48:46.824916 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 14838
I20260806 02:48:46.849100 1865 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 02:48:47.195837 1865 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 02:48:47.196113 1865 mini_ranger.cc:66] Stopping Ranger...
I20260806 02:48:47.543349 1865 mini_ranger.cc:68] Stopped Ranger
2026-08-06 02:48:47.543 UTC [13883] LOG: received smart shutdown request
2026-08-06 02:48:47.549 UTC [13883] LOG: background worker "logical replication launcher" (PID 13892) exited with exit code 1
2026-08-06 02:48:47.550 UTC [13887] LOG: shutting down
2026-08-06 02:48:47.769 UTC [13883] LOG: database system is shut down
2026-08-06T02:48:47Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (59291 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:48:48 dist-test-slave-8tdl krb5kdc[15420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:48:48 dist-test-slave-8tdl krb5kdc[15420](info): set up 2 sockets
Aug 06 02:48:48 dist-test-slave-8tdl krb5kdc[15420](info): commencing operation
krb5kdc: starting...
W20260806 02:48:49.074440 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:48:49 dist-test-slave-8tdl krb5kdc[15420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:48:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:48:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
I20260806 02:48:49.359536 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:39673
--webserver_interface=127.1.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:35051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:39673,127.1.210.125:34911,127.1.210.124:44299
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:48:49.715167 15433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:49.715482 15433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:49.715644 15433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:49.725325 15433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:48:49.725495 15433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:49.725584 15433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:49.725687 15433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:48:49.725754 15433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:48:49.737649 15433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:35051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:39673,127.1.210.125:34911,127.1.210.124:44299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:39673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:48:49.739243 15433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:49.740609 15433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:49.753904 15439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:49.753856 15440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:49.753816 15442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:49.755007 15433 server_base.cc:1034] running on GCE node
I20260806 02:48:49.755945 15433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:49.757500 15433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:49.758801 15433 hybrid_clock.cc:648] HybridClock initialized: now 1785984529758754 us; error 37 us; skew 500 ppm
I20260806 02:48:49.759235 15433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:48:49 dist-test-slave-8tdl krb5kdc[15420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984529, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:49.788287 15433 init.cc:377] Logged in from keytab as kudu/127.1.210.126@KRBTEST.COM (short username kudu)
I20260806 02:48:49.790253 15433 webserver.cc:460] Webserver started at http://127.1.210.126:45753/ using document root <none> and password file <none>
I20260806 02:48:49.791097 15433 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:49.791275 15433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:49.791620 15433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:48:49.794962 15433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "0554fa6958dc4c489f2346d91579428e"
format_stamp: "Formatted at 2026-08-06 02:48:49 on dist-test-slave-8tdl"
server_key: "5a2d55163696a2ea9ad29f53947306dc"
server_key_iv: "8dde68c08dac456db12057ae1b34ced2"
server_key_version: "encryptionkey@0"
I20260806 02:48:49.795897 15433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "0554fa6958dc4c489f2346d91579428e"
format_stamp: "Formatted at 2026-08-06 02:48:49 on dist-test-slave-8tdl"
server_key: "5a2d55163696a2ea9ad29f53947306dc"
server_key_iv: "8dde68c08dac456db12057ae1b34ced2"
server_key_version: "encryptionkey@0"
I20260806 02:48:49.801739 15433 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 02:48:49.806056 15449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:49.807780 15433 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:48:49.808009 15433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "0554fa6958dc4c489f2346d91579428e"
format_stamp: "Formatted at 2026-08-06 02:48:49 on dist-test-slave-8tdl"
server_key: "5a2d55163696a2ea9ad29f53947306dc"
server_key_iv: "8dde68c08dac456db12057ae1b34ced2"
server_key_version: "encryptionkey@0"
I20260806 02:48:49.808254 15433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:49.822387 15433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:49.827420 15433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:49.827729 15433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:49.849020 15433 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:39673
I20260806 02:48:49.849076 15500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:39673 every 8 connection(s)
I20260806 02:48:49.850647 15433 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:48:49.854564 15501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:48:49.857995 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 15433
I20260806 02:48:49.858253 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:48:49.859155 1865 external_mini_cluster.cc:1428] Setting key 70077f3c1cbc88c0b0f8b579be592cf6
I20260806 02:48:49.864184 15501 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39673 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:48:49 dist-test-slave-8tdl krb5kdc[15420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984529, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.210.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.125@KRBTEST.COM" created.
I20260806 02:48:49.890471 15503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:49.868875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:39673 (local address 127.0.0.1:53766)
0806 02:48:49.871122 (+ 2247us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:49.871186 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:49.873129 (+ 1943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:49.874058 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:49.874065 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:49.874906 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:49.875275 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:49.875292 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:49.876881 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:49.876885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:49.878002 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:49.878014 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:49.879080 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:49.879088 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:49.879267 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:49.879980 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:49.880016 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:49.882396 (+ 2380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:49.886021 (+ 3625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:49.886034 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:49.886055 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:49.886560 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:49.887003 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:49.887010 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:49.887015 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:49.887210 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:49.888291 (+ 1081us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:49.888298 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:49.888518 (+ 220us) client_negotiation.cc:769] Sending connection context
0806 02:48:49.888777 (+ 259us) client_negotiation.cc:240] Negotiation successful
0806 02:48:49.889252 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"thread_start_us":466,"threads_started":1}
I20260806 02:48:49.890506 15504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:49.868875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53766 (local address 127.1.210.126:39673)
0806 02:48:49.871186 (+ 2311us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:49.871202 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:49.871333 (+ 131us) server_negotiation.cc:407] Connection header received
0806 02:48:49.873551 (+ 2218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:49.873556 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:49.873811 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:49.874023 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:49.875478 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:49.876712 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:49.878160 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:49.878818 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:49.879014 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:49.882643 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:49.882652 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:49.882666 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:49.882740 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:49.885878 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:49.886725 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:49.886732 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:49.886734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:49.886877 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:49.887303 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:49.887311 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:49.887314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:49.888066 (+ 752us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:49.888289 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:49.888965 (+ 676us) server_negotiation.cc:299] Negotiation successful
0806 02:48:49.889495 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":152,"server-negotiator.queue_time_us":2106,"spinlock_wait_cycles":71936,"thread_start_us":1746,"threads_started":1}
I20260806 02:48:49.893988 15501 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.125" port: 34911 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:48:49.895412 15503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:49.894934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:34911 (local address 127.0.0.1:58850)
0806 02:48:49.895170 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:49.895283 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.125:34911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260806 02:48:49.895726 15452 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.210.125:34911: connect: Connection refused (error 111)
I20260806 02:48:49.897405 15503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:49.897047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:34911 (local address 127.0.0.1:58860)
0806 02:48:49.897199 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:49.897275 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.125:34911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 02:48:49.897620 15501 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.125:34911: Network error: Client connection negotiation failed: client connection to 127.1.210.125:34911: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.125.keytab.
Entry for principal kudu/127.1.210.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.125.keytab.
I20260806 02:48:49.923491 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.125:34911
--webserver_interface=127.1.210.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.125.keytab
--principal=kudu/127.1.210.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:35051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:39673,127.1.210.125:34911,127.1.210.124:44299
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:48:49.947075 15501 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.125" port: 34911 } attempt: 1
I20260806 02:48:49.948524 15503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:49.947893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:34911 (local address 127.0.0.1:58874)
0806 02:48:49.948150 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:49.948296 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.125:34911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260806 02:48:49.949824 15503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:49.949311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:34911 (local address 127.0.0.1:58884)
0806 02:48:49.949510 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:49.949617 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.125:34911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 02:48:49.950089 15501 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.125:34911: Network error: Client connection negotiation failed: client connection to 127.1.210.125:34911: connect: Connection refused (error 111)
I20260806 02:48:50.018391 15501 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.125" port: 34911 } attempt: 2
I20260806 02:48:50.019840 15503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:50.019235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:34911 (local address 127.0.0.1:58894)
0806 02:48:50.019525 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:50.019655 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.125:34911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260806 02:48:50.021255 15503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:50.020657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:34911 (local address 127.0.0.1:58896)
0806 02:48:50.020845 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:50.021035 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.125:34911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260806 02:48:50.021642 15501 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.125:34911: Network error: Client connection negotiation failed: client connection to 127.1.210.125:34911: connect: Connection refused (error 111)
I20260806 02:48:50.112927 15501 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.125" port: 34911 } attempt: 3
I20260806 02:48:50.114202 15503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:50.113633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:34911 (local address 127.0.0.1:58910)
0806 02:48:50.113853 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:50.113987 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.125:34911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260806 02:48:50.115407 15503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:50.114975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:34911 (local address 127.0.0.1:58912)
0806 02:48:50.115131 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:50.115237 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.125:34911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 02:48:50.115636 15501 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.125:34911: Network error: Client connection negotiation failed: client connection to 127.1.210.125:34911: connect: Connection refused (error 111)
I20260806 02:48:50.258924 15501 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.125" port: 34911 } attempt: 4
I20260806 02:48:50.260483 15503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:50.259734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:34911 (local address 127.0.0.1:58922)
0806 02:48:50.260110 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:50.260256 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.125:34911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195}
I20260806 02:48:50.261813 15503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:50.261250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:34911 (local address 127.0.0.1:58932)
0806 02:48:50.261493 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:50.261630 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.125:34911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 02:48:50.262055 15501 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.125:34911: Network error: Client connection negotiation failed: client connection to 127.1.210.125:34911: connect: Connection refused (error 111)
W20260806 02:48:50.300688 15508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:50.301015 15508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:50.301123 15508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:50.309178 15508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:48:50.309321 15508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:50.309372 15508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:50.309434 15508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:48:50.309481 15508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:48:50.320396 15508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:35051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:39673,127.1.210.125:34911,127.1.210.124:44299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.125.keytab
--principal=kudu/127.1.210.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.125:34911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.210.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:48:50.321841 15508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:50.323237 15508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:50.336670 15513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:50.336709 15514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:50.336755 15516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:50.337589 15508 server_base.cc:1034] running on GCE node
I20260806 02:48:50.338500 15508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:50.340155 15508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:50.341382 15508 hybrid_clock.cc:648] HybridClock initialized: now 1785984530341310 us; error 47 us; skew 500 ppm
I20260806 02:48:50.341806 15508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:48:50 dist-test-slave-8tdl krb5kdc[15420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984530, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:50.345368 15508 init.cc:377] Logged in from keytab as kudu/127.1.210.125@KRBTEST.COM (short username kudu)
I20260806 02:48:50.347139 15508 webserver.cc:460] Webserver started at http://127.1.210.125:36659/ using document root <none> and password file <none>
I20260806 02:48:50.347929 15508 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:50.348079 15508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:50.348395 15508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:48:50.351780 15508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-1/data/instance:
uuid: "a9912610f343455bab6c9248f5d4a246"
format_stamp: "Formatted at 2026-08-06 02:48:50 on dist-test-slave-8tdl"
server_key: "00f476a6875dd628935d106fa08f210e"
server_key_iv: "bec3c6fbde68d56a4bbdd8c5f3cbbaab"
server_key_version: "encryptionkey@0"
I20260806 02:48:50.352674 15508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-1/wal/instance:
uuid: "a9912610f343455bab6c9248f5d4a246"
format_stamp: "Formatted at 2026-08-06 02:48:50 on dist-test-slave-8tdl"
server_key: "00f476a6875dd628935d106fa08f210e"
server_key_iv: "bec3c6fbde68d56a4bbdd8c5f3cbbaab"
server_key_version: "encryptionkey@0"
I20260806 02:48:50.358470 15508 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 02:48:50.362371 15523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:50.364073 15508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:48:50.364301 15508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-1/wal
uuid: "a9912610f343455bab6c9248f5d4a246"
format_stamp: "Formatted at 2026-08-06 02:48:50 on dist-test-slave-8tdl"
server_key: "00f476a6875dd628935d106fa08f210e"
server_key_iv: "bec3c6fbde68d56a4bbdd8c5f3cbbaab"
server_key_version: "encryptionkey@0"
I20260806 02:48:50.364537 15508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:50.402182 15508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:50.407070 15508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:50.407388 15508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:50.427472 15508 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.125:34911
I20260806 02:48:50.427501 15575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.125:34911 every 8 connection(s)
I20260806 02:48:50.428956 15508 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-1/data/info.pb
I20260806 02:48:50.432964 15576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:48:50.433410 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 15508
I20260806 02:48:50.433635 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-1/wal/instance
I20260806 02:48:50.434240 1865 external_mini_cluster.cc:1428] Setting key 2ade5c8cad77fc02b9773a458aa50b24
I20260806 02:48:50.443995 15576 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39673 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:48:50 dist-test-slave-8tdl krb5kdc[15420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984530, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.125@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.210.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.124@KRBTEST.COM" created.
I20260806 02:48:50.467458 15579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:50.447339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53770 (local address 127.1.210.126:39673)
0806 02:48:50.447903 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:50.447907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:50.448963 (+ 1056us) server_negotiation.cc:407] Connection header received
0806 02:48:50.451178 (+ 2215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:50.451184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:50.451280 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:50.451396 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:50.453039 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:50.453863 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:50.455615 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:50.456414 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:50.456593 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:50.460282 (+ 3689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:50.460287 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:50.460291 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:50.460332 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:50.463605 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:50.464758 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:50.464763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:50.464765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:50.464898 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:50.465442 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:50.465445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:50.465447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:50.465764 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:50.465882 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:50.466917 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0806 02:48:50.467164 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":325,"threads_started":1}
I20260806 02:48:50.469164 15580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:50.447938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:39673 (local address 127.0.0.1:53770)
0806 02:48:50.448788 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:50.448829 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:50.450871 (+ 2042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:50.451662 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:50.451671 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:50.452255 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:50.452859 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:50.452877 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:50.454043 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:50.454048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:50.455405 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:50.455419 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:50.456658 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:50.456664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:50.456873 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:50.457638 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:50.457663 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:50.459996 (+ 2333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:50.463777 (+ 3781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:50.463790 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:50.463813 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:50.464590 (+ 777us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:50.465017 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:50.465024 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:50.465030 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:50.465324 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:50.465897 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:50.465908 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:50.466460 (+ 552us) client_negotiation.cc:769] Sending connection context
0806 02:48:50.466821 (+ 361us) client_negotiation.cc:240] Negotiation successful
0806 02:48:50.467473 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":365,"threads_started":1}
I20260806 02:48:50.471330 15576 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.125" port: 34911 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:48:50 dist-test-slave-8tdl krb5kdc[15420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984530, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.125@KRBTEST.COM for kudu/127.1.210.125@KRBTEST.COM
I20260806 02:48:50.487260 15580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:50.472203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:34911 (local address 127.0.0.1:58934)
0806 02:48:50.472435 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:50.472461 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:50.472589 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:50.473621 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:50.473625 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:50.474149 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:50.474280 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:50.474289 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:50.475736 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:50.475744 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:50.476601 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:50.476612 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:50.477439 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:50.477445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:50.477656 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:50.478299 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:50.478317 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:50.480513 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:50.484250 (+ 3737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:50.484256 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:50.484260 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:50.484657 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:50.485309 (+ 652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:50.485315 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:50.485318 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:50.485490 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:50.486177 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:50.486184 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:50.486341 (+ 157us) client_negotiation.cc:769] Sending connection context
0806 02:48:50.486484 (+ 143us) client_negotiation.cc:240] Negotiation successful
0806 02:48:50.486815 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260806 02:48:50.487320 15581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:50.472404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58934 (local address 127.1.210.125:34911)
0806 02:48:50.473187 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:50.473193 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:50.473220 (+ 27us) server_negotiation.cc:407] Connection header received
0806 02:48:50.473264 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:50.473270 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:50.473461 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:50.473632 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:50.474435 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:50.475604 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:50.476741 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:50.477291 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:50.477432 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:50.480741 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:50.480755 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:50.480769 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:50.480812 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:50.484098 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:50.484811 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:50.484819 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:50.484821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:50.485051 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:50.485636 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:50.485640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:50.485642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:50.486034 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:50.486282 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:50.486638 (+ 356us) server_negotiation.cc:299] Negotiation successful
0806 02:48:50.486941 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":635,"thread_start_us":483,"threads_started":1}
I20260806 02:48:50.489909 15576 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44299 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:48:50.491158 15580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:50.490640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:44299 (local address 127.0.0.1:36550)
0806 02:48:50.490847 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:50.490988 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260806 02:48:50.491425 15524 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.210.124:44299: connect: Connection refused (error 111)
I20260806 02:48:50.493132 15580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:50.492755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:44299 (local address 127.0.0.1:36552)
0806 02:48:50.492902 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:50.493015 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 02:48:50.493348 15576 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.124:44299: Network error: Client connection negotiation failed: client connection to 127.1.210.124:44299: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.124.keytab.
Entry for principal kudu/127.1.210.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.124.keytab.
I20260806 02:48:50.498889 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.124:44299
--webserver_interface=127.1.210.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.124.keytab
--principal=kudu/127.1.210.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:35051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:39673,127.1.210.125:34911,127.1.210.124:44299
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:48:50.542771 15576 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44299 } attempt: 1
I20260806 02:48:50.544095 15580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:50.543579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:44299 (local address 127.0.0.1:36562)
0806 02:48:50.543779 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:50.543888 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260806 02:48:50.545089 15580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:50.544737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:44299 (local address 127.0.0.1:36576)
0806 02:48:50.544901 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:50.544972 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 02:48:50.545295 15576 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.124:44299: Network error: Client connection negotiation failed: client connection to 127.1.210.124:44299: connect: Connection refused (error 111)
I20260806 02:48:50.561331 15501 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.125" port: 34911 } attempt: 5
Aug 06 02:48:50 dist-test-slave-8tdl krb5kdc[15420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984529, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for kudu/127.1.210.125@KRBTEST.COM
I20260806 02:48:50.577467 15503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:50.562162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:34911 (local address 127.0.0.1:58946)
0806 02:48:50.562448 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:50.562473 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:50.562617 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:50.563255 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:50.563260 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:50.563670 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:50.563843 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:50.563854 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:50.565314 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:50.565319 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:50.566287 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:50.566301 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:50.567370 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:50.567374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:50.567531 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:50.568190 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:50.568214 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:50.570844 (+ 2630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:50.574961 (+ 4117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:50.574966 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:50.574969 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:50.575379 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:50.575832 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:50.575835 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:50.575837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:50.575995 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:50.576659 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:50.576663 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:50.576810 (+ 147us) client_negotiation.cc:769] Sending connection context
0806 02:48:50.576928 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 02:48:50.577197 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260806 02:48:50.577798 15581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:50.562455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58946 (local address 127.1.210.125:34911)
0806 02:48:50.562703 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:50.562711 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:50.562735 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:48:50.562822 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:50.562827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:50.562911 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:50.563195 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:50.564025 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:50.565145 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:50.566442 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:50.567216 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:50.567352 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:50.571066 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:50.571074 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:50.571079 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:50.571122 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:50.574760 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:50.575516 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:50.575522 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:50.575526 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:50.575687 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:50.576102 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:50.576106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:50.576110 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:50.576514 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:50.576657 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:50.577112 (+ 455us) server_negotiation.cc:299] Negotiation successful
0806 02:48:50.577426 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 02:48:50.578881 15501 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44299 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:48:50.580159 15503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:50.579692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:44299 (local address 127.0.0.1:36578)
0806 02:48:50.579864 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:50.579950 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260806 02:48:50.581110 15503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:50.580759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:44299 (local address 127.0.0.1:36584)
0806 02:48:50.580919 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:50.581013 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 02:48:50.581319 15501 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.124:44299: Network error: Client connection negotiation failed: client connection to 127.1.210.124:44299: connect: Connection refused (error 111)
I20260806 02:48:50.613534 15576 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44299 } attempt: 2
I20260806 02:48:50.614938 15580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:50.614345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:44299 (local address 127.0.0.1:36598)
0806 02:48:50.614568 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:50.614723 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260806 02:48:50.615968 15580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:50.615618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:44299 (local address 127.0.0.1:36602)
0806 02:48:50.615764 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:50.615843 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 02:48:50.616178 15576 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.124:44299: Network error: Client connection negotiation failed: client connection to 127.1.210.124:44299: connect: Connection refused (error 111)
I20260806 02:48:50.632550 15501 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44299 } attempt: 1
I20260806 02:48:50.633852 15503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:50.633296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:44299 (local address 127.0.0.1:36606)
0806 02:48:50.633533 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:50.633647 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260806 02:48:50.634918 15503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:50.634524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:44299 (local address 127.0.0.1:36622)
0806 02:48:50.634670 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:50.634793 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 02:48:50.635151 15501 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.124:44299: Network error: Client connection negotiation failed: client connection to 127.1.210.124:44299: connect: Connection refused (error 111)
I20260806 02:48:50.703382 15501 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44299 } attempt: 2
I20260806 02:48:50.704658 15503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:50.704141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:44299 (local address 127.0.0.1:36632)
0806 02:48:50.704376 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:50.704483 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260806 02:48:50.705688 15503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:50.705327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:44299 (local address 127.0.0.1:36648)
0806 02:48:50.705491 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:50.705563 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 02:48:50.705897 15501 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.124:44299: Network error: Client connection negotiation failed: client connection to 127.1.210.124:44299: connect: Connection refused (error 111)
I20260806 02:48:50.707391 15576 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44299 } attempt: 3
I20260806 02:48:50.708451 15580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:50.708013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:44299 (local address 127.0.0.1:36658)
0806 02:48:50.708216 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:50.708303 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 02:48:50.709510 15580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:50.709087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:44299 (local address 127.0.0.1:36662)
0806 02:48:50.709313 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:50.709397 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260806 02:48:50.709694 15576 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.124:44299: Network error: Client connection negotiation failed: client connection to 127.1.210.124:44299: connect: Connection refused (error 111)
I20260806 02:48:50.812132 15501 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44299 } attempt: 3
I20260806 02:48:50.813315 15503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:50.812833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:44299 (local address 127.0.0.1:36664)
0806 02:48:50.813047 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:50.813145 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260806 02:48:50.814484 15503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:50.814058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:44299 (local address 127.0.0.1:36678)
0806 02:48:50.814293 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:50.814368 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260806 02:48:50.814706 15501 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.124:44299: Network error: Client connection negotiation failed: client connection to 127.1.210.124:44299: connect: Connection refused (error 111)
I20260806 02:48:50.852927 15576 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44299 } attempt: 4
I20260806 02:48:50.854254 15580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:50.853687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:44299 (local address 127.0.0.1:36690)
0806 02:48:50.853937 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:50.854074 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260806 02:48:50.855423 15580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:50.855040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:44299 (local address 127.0.0.1:36700)
0806 02:48:50.855239 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:50.855312 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260806 02:48:50.855723 15576 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.124:44299: Network error: Client connection negotiation failed: client connection to 127.1.210.124:44299: connect: Connection refused (error 111)
W20260806 02:48:50.865849 15584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:50.866168 15584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:50.866258 15584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:50.873835 15584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:48:50.873958 15584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:50.874006 15584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:50.874079 15584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:48:50.874125 15584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:48:50.884572 15584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:35051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:39673,127.1.210.125:34911,127.1.210.124:44299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.124.keytab
--principal=kudu/127.1.210.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.124:44299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.210.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:48:50.885962 15584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:50.887398 15584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:50.899791 15592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:50.899962 15589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:50.899958 15590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:50.900738 15584 server_base.cc:1034] running on GCE node
I20260806 02:48:50.901600 15584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:50.903208 15584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:50.904433 15584 hybrid_clock.cc:648] HybridClock initialized: now 1785984530904374 us; error 50 us; skew 500 ppm
I20260806 02:48:50.904789 15584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:48:50 dist-test-slave-8tdl krb5kdc[15420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984530, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:50.908377 15584 init.cc:377] Logged in from keytab as kudu/127.1.210.124@KRBTEST.COM (short username kudu)
I20260806 02:48:50.910202 15584 webserver.cc:460] Webserver started at http://127.1.210.124:38525/ using document root <none> and password file <none>
I20260806 02:48:50.911010 15584 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:50.911134 15584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:50.911401 15584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:48:50.914662 15584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-2/data/instance:
uuid: "757edc90130848cd8e9d696d5cc84e0b"
format_stamp: "Formatted at 2026-08-06 02:48:50 on dist-test-slave-8tdl"
server_key: "1ab596880283ee2aace146560aadc9f5"
server_key_iv: "dbfe58c9cba9b0a9fef36a674cba2823"
server_key_version: "encryptionkey@0"
I20260806 02:48:50.915603 15584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-2/wal/instance:
uuid: "757edc90130848cd8e9d696d5cc84e0b"
format_stamp: "Formatted at 2026-08-06 02:48:50 on dist-test-slave-8tdl"
server_key: "1ab596880283ee2aace146560aadc9f5"
server_key_iv: "dbfe58c9cba9b0a9fef36a674cba2823"
server_key_version: "encryptionkey@0"
I20260806 02:48:50.921478 15584 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 02:48:50.925390 15599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:50.927093 15584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 02:48:50.927289 15584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-2/wal
uuid: "757edc90130848cd8e9d696d5cc84e0b"
format_stamp: "Formatted at 2026-08-06 02:48:50 on dist-test-slave-8tdl"
server_key: "1ab596880283ee2aace146560aadc9f5"
server_key_iv: "dbfe58c9cba9b0a9fef36a674cba2823"
server_key_version: "encryptionkey@0"
I20260806 02:48:50.927508 15584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:50.945689 15584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:50.950789 15584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:50.951097 15584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:50.971108 15584 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.124:44299
I20260806 02:48:50.971194 15651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.124:44299 every 8 connection(s)
I20260806 02:48:50.972584 15584 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-2/data/info.pb
I20260806 02:48:50.976768 15652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:48:50.977651 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 15584
I20260806 02:48:50.977864 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984304136196-1865-0/minicluster-data/master-2/wal/instance
I20260806 02:48:50.978331 1865 external_mini_cluster.cc:1428] Setting key 309fbca228a9c40086cb6c7c2087e3df
I20260806 02:48:50.986790 15652 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39673 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:48:50.991946 15501 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44299 } attempt: 4
Aug 06 02:48:50 dist-test-slave-8tdl krb5kdc[15420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.124@KRBTEST.COM
Aug 06 02:48:50 dist-test-slave-8tdl krb5kdc[15420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984530, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.124@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
Aug 06 02:48:50 dist-test-slave-8tdl krb5kdc[15420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984529, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for kudu/127.1.210.124@KRBTEST.COM
I20260806 02:48:51.008518 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:50.980240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36708 (local address 127.1.210.124:44299)
0806 02:48:50.981089 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:50.981102 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:50.981145 (+ 43us) server_negotiation.cc:407] Connection header received
0806 02:48:50.983056 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:50.983078 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:50.983525 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:50.983895 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:50.986040 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:50.987564 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:50.989730 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:50.991040 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:50.991252 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:50.996756 (+ 5504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:50.996772 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:50.996791 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:50.996849 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:51.002277 (+ 5428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:51.003386 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:51.003399 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:51.003412 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:51.003664 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:51.004264 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:51.004270 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:51.004274 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:51.004972 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:51.005622 (+ 650us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:51.005976 (+ 354us) server_negotiation.cc:299] Negotiation successful
0806 02:48:51.006535 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":385,"threads_started":1}
I20260806 02:48:51.009013 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:50.990095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:39673 (local address 127.0.0.1:53786)
0806 02:48:50.991286 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:50.991333 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:50.991535 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:50.992065 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:50.992072 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:50.993035 (+ 963us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:50.993547 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:50.993565 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:50.995162 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:50.995169 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:50.996865 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:50.996875 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:50.997761 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:50.997767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:50.997943 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:50.998563 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:50.998580 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:51.001285 (+ 2705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:51.005321 (+ 4036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:51.005333 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:51.005340 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:51.005807 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:51.006457 (+ 650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:51.006463 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:51.006468 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:51.006666 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:51.007466 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:51.007478 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:51.007751 (+ 273us) client_negotiation.cc:769] Sending connection context
0806 02:48:51.007983 (+ 232us) client_negotiation.cc:240] Negotiation successful
0806 02:48:51.008545 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"thread_start_us":700,"threads_started":1}
I20260806 02:48:51.009508 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:50.990124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53786 (local address 127.1.210.126:39673)
0806 02:48:50.990863 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:50.990869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:50.991544 (+ 675us) server_negotiation.cc:407] Connection header received
0806 02:48:50.991704 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:50.991709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:50.991815 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:50.991964 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:50.993739 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:50.994947 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:50.997052 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:50.997573 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:50.997721 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:51.001496 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:51.001501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:51.001505 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:51.001543 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:51.005174 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:51.005937 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:51.005945 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:51.005950 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:51.006127 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:51.006820 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:51.006827 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:51.006832 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:51.007307 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:51.007482 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:51.008306 (+ 824us) server_negotiation.cc:299] Negotiation successful
0806 02:48:51.008580 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":579,"thread_start_us":350,"threads_started":1}
I20260806 02:48:51.011158 15652 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.125" port: 34911 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:48:51.012108 15503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:50.992558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:44299 (local address 127.0.0.1:36716)
0806 02:48:50.992857 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:50.992883 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:50.993114 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:50.994160 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:50.994166 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:50.994703 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:50.994983 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:50.995001 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:50.996708 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:50.996715 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:50.999106 (+ 2391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:50.999133 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:50.999933 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:50.999941 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:51.000182 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:51.000772 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:51.000805 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:51.004060 (+ 3255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:51.009134 (+ 5074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:51.009140 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:51.009143 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:51.009774 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:51.010178 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:51.010183 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:51.010187 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:51.010344 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:51.011060 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:51.011064 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:51.011263 (+ 199us) client_negotiation.cc:769] Sending connection context
0806 02:48:51.011399 (+ 136us) client_negotiation.cc:240] Negotiation successful
0806 02:48:51.011716 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":2}
I20260806 02:48:51.013281 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:50.992642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36716 (local address 127.1.210.124:44299)
0806 02:48:50.993796 (+ 1154us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:50.993804 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:50.993828 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:48:50.993880 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:50.993886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:50.993963 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:50.994106 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:50.995237 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:50.996560 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:50.999278 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:50.999768 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:50.999915 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:51.004316 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:51.004323 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:51.004327 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:51.004369 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:51.008956 (+ 4587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:51.009927 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:51.009932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:51.009934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:51.010058 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:51.010479 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:51.010484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:51.010487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:51.010908 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:51.012461 (+ 1553us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:51.012599 (+ 138us) server_negotiation.cc:299] Negotiation successful
0806 02:48:51.012838 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1023,"thread_start_us":815,"threads_started":1}
Aug 06 02:48:51 dist-test-slave-8tdl krb5kdc[15420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984530, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.124@KRBTEST.COM for kudu/127.1.210.125@KRBTEST.COM
I20260806 02:48:51.020248 15501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0554fa6958dc4c489f2346d91579428e: Bootstrap starting.
Aug 06 02:48:51 dist-test-slave-8tdl krb5kdc[15420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:48:51.024844 15501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0554fa6958dc4c489f2346d91579428e: Neither blocks nor log segments found. Creating new log.
I20260806 02:48:51.026106 15501 log.cc:824] T 00000000000000000000000000000000 P 0554fa6958dc4c489f2346d91579428e: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:51.026590 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:51.012077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:34911 (local address 127.0.0.1:58948)
0806 02:48:51.012261 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:51.012282 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:51.012380 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:51.012998 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:51.013003 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:51.013545 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:51.013721 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:51.013730 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:51.015110 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:51.015117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:51.016009 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:51.016021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:51.016751 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:51.016757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:51.016982 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:51.017575 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:51.017593 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:51.020029 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:51.023439 (+ 3410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:51.023445 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:51.023449 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:51.023853 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:51.024355 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:51.024364 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:51.024370 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:51.024558 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:51.025538 (+ 980us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:51.025543 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:51.025733 (+ 190us) client_negotiation.cc:769] Sending connection context
0806 02:48:51.025888 (+ 155us) client_negotiation.cc:240] Negotiation successful
0806 02:48:51.026222 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260806 02:48:51.026788 15581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:51.012133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58948 (local address 127.1.210.125:34911)
0806 02:48:51.012339 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:51.012347 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:51.012369 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:48:51.012606 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:51.012613 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:51.012696 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:51.012771 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:51.013879 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:51.014970 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:51.016157 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:51.016596 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:51.016735 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:51.020229 (+ 3494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:51.020234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:51.020238 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:51.020274 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:51.023287 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:51.023984 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:51.023990 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:51.023998 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:51.024206 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:51.024696 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:51.024702 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:51.024705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:51.025096 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:51.025420 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:51.026121 (+ 701us) server_negotiation.cc:299] Negotiation successful
0806 02:48:51.026350 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 02:48:51.027840 15652 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44299 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:48:51.030388 15501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0554fa6958dc4c489f2346d91579428e: No bootstrap required, opened a new log
I20260806 02:48:51.035005 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:51.012722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53802 (local address 127.1.210.126:39673)
0806 02:48:51.012893 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:51.012900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:51.013286 (+ 386us) server_negotiation.cc:407] Connection header received
0806 02:48:51.013522 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:51.013527 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:51.013608 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:51.014785 (+ 1177us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:51.015481 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:51.016624 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:51.018968 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:51.019692 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:51.019827 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:51.026056 (+ 6229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:51.026065 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:51.026072 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:51.026126 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:51.030654 (+ 4528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:51.031665 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:51.031670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:51.031674 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:51.031845 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:51.032353 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:51.032359 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:51.032362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:51.032825 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:51.034143 (+ 1318us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:51.034301 (+ 158us) server_negotiation.cc:299] Negotiation successful
0806 02:48:51.034576 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Aug 06 02:48:51 dist-test-slave-8tdl krb5kdc[15420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984530, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.124@KRBTEST.COM for kudu/127.1.210.124@KRBTEST.COM
I20260806 02:48:51.042557 15501 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0554fa6958dc4c489f2346d91579428e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0554fa6958dc4c489f2346d91579428e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39673 } } peers { permanent_uuid: "a9912610f343455bab6c9248f5d4a246" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 34911 } } peers { permanent_uuid: "757edc90130848cd8e9d696d5cc84e0b" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44299 } }
I20260806 02:48:51.042987 15501 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0554fa6958dc4c489f2346d91579428e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:48:51.043103 15501 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0554fa6958dc4c489f2346d91579428e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0554fa6958dc4c489f2346d91579428e, State: Initialized, Role: FOLLOWER
I20260806 02:48:51.043876 15501 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0554fa6958dc4c489f2346d91579428e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0554fa6958dc4c489f2346d91579428e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39673 } } peers { permanent_uuid: "a9912610f343455bab6c9248f5d4a246" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 34911 } } peers { permanent_uuid: "757edc90130848cd8e9d696d5cc84e0b" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44299 } }
I20260806 02:48:51.045207 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:51.028626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:44299 (local address 127.0.0.1:36728)
0806 02:48:51.029041 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:51.029064 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:51.029167 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:51.029510 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:51.029514 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:51.029915 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:51.030061 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:51.030069 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:51.031355 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:51.031361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:51.032635 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:51.032651 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:51.033431 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:51.033436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:51.033633 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:51.034295 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:51.034328 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:51.037476 (+ 3148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:51.041483 (+ 4007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:51.041488 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:51.041492 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:51.042082 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:51.043129 (+ 1047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:51.043134 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:51.043138 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:51.043310 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:51.044073 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:51.044079 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:51.044248 (+ 169us) client_negotiation.cc:769] Sending connection context
0806 02:48:51.044452 (+ 204us) client_negotiation.cc:240] Negotiation successful
0806 02:48:51.044853 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269}
I20260806 02:48:51.045030 15660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0554fa6958dc4c489f2346d91579428e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0554fa6958dc4c489f2346d91579428e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39673 } } peers { permanent_uuid: "a9912610f343455bab6c9248f5d4a246" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 34911 } } peers { permanent_uuid: "757edc90130848cd8e9d696d5cc84e0b" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44299 } } }
I20260806 02:48:51.045895 15660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0554fa6958dc4c489f2346d91579428e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:48:51.046062 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:51.029084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36728 (local address 127.1.210.124:44299)
0806 02:48:51.029247 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:51.029255 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:51.029271 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:48:51.029310 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:51.029320 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:51.029392 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:51.029490 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:51.030207 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:51.031172 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:51.032754 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:51.033264 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:51.033399 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:51.037686 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:51.037692 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:51.037697 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:51.037734 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:51.041342 (+ 3608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:51.042798 (+ 1456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:51.042805 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:51.042810 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:51.042986 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:51.043456 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:51.043464 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:51.043470 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:51.043909 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:51.044089 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:51.045516 (+ 1427us) server_negotiation.cc:299] Negotiation successful
0806 02:48:51.045748 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:48:51.046486 15501 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0554fa6958dc4c489f2346d91579428e [sys.catalog]: configured and running, proceeding with master startup.
Aug 06 02:48:51 dist-test-slave-8tdl krb5kdc[15420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.125@KRBTEST.COM
I20260806 02:48:51.051774 15652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 757edc90130848cd8e9d696d5cc84e0b: Bootstrap starting.
W20260806 02:48:51.056862 15671 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0554fa6958dc4c489f2346d91579428e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:48:51.057159 15671 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:48:51.059592 15652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 757edc90130848cd8e9d696d5cc84e0b: Neither blocks nor log segments found. Creating new log.
I20260806 02:48:51.061048 15652 log.cc:824] T 00000000000000000000000000000000 P 757edc90130848cd8e9d696d5cc84e0b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:51.062528 15581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:51.036879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58964 (local address 127.1.210.125:34911)
0806 02:48:51.037055 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:51.037061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:51.037287 (+ 226us) server_negotiation.cc:407] Connection header received
0806 02:48:51.037345 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:51.037349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:51.037420 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:51.037525 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:51.038973 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:51.039891 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:51.041384 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:51.042069 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:51.042211 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:51.052109 (+ 9898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:51.052119 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:51.052124 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:51.052175 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:51.056018 (+ 3843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:51.059732 (+ 3714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:51.059740 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:51.059744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:51.059937 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:51.060483 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:51.060489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:51.060492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:51.060889 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:51.061162 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:51.061857 (+ 695us) server_negotiation.cc:299] Negotiation successful
0806 02:48:51.062134 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 02:48:51.071462 1865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:48:51.071940 15652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 757edc90130848cd8e9d696d5cc84e0b: No bootstrap required, opened a new log
I20260806 02:48:51.057029 15433 master_runner.cc:186] Error getting master registration for 127.1.210.125:34911: OK, instance_id { permanent_uuid: "a9912610f343455bab6c9248f5d4a246" instance_seqno: 1785984530421696 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 02:48:51.073433 15433 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260806 02:48:51.080708 15455 tablet.cc:2369] T 00000000000000000000000000000000 P 0554fa6958dc4c489f2346d91579428e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:48:51.085090 15652 raft_consensus.cc:348] T 00000000000000000000000000000000 P 757edc90130848cd8e9d696d5cc84e0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0554fa6958dc4c489f2346d91579428e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39673 } } peers { permanent_uuid: "a9912610f343455bab6c9248f5d4a246" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 34911 } } peers { permanent_uuid: "757edc90130848cd8e9d696d5cc84e0b" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44299 } }
I20260806 02:48:51.085718 15652 raft_consensus.cc:374] T 00000000000000000000000000000000 P 757edc90130848cd8e9d696d5cc84e0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:48:51.085886 15652 raft_consensus.cc:729] T 00000000000000000000000000000000 P 757edc90130848cd8e9d696d5cc84e0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 757edc90130848cd8e9d696d5cc84e0b, State: Initialized, Role: FOLLOWER
I20260806 02:48:51.086969 15652 consensus_queue.cc:260] T 00000000000000000000000000000000 P 757edc90130848cd8e9d696d5cc84e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0554fa6958dc4c489f2346d91579428e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39673 } } peers { permanent_uuid: "a9912610f343455bab6c9248f5d4a246" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 34911 } } peers { permanent_uuid: "757edc90130848cd8e9d696d5cc84e0b" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44299 } }
I20260806 02:48:51.096648 15652 sys_catalog.cc:564] T 00000000000000000000000000000000 P 757edc90130848cd8e9d696d5cc84e0b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:48:51.094554 15680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 757edc90130848cd8e9d696d5cc84e0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0554fa6958dc4c489f2346d91579428e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39673 } } peers { permanent_uuid: "a9912610f343455bab6c9248f5d4a246" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 34911 } } peers { permanent_uuid: "757edc90130848cd8e9d696d5cc84e0b" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44299 } } }
I20260806 02:48:51.098390 15680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 757edc90130848cd8e9d696d5cc84e0b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:48:51.120671 15581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:51.078105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58968 (local address 127.1.210.125:34911)
0806 02:48:51.078293 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:51.078300 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:51.090499 (+ 12199us) server_negotiation.cc:407] Connection header received
0806 02:48:51.090782 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:51.090787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:51.090873 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:51.090967 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:51.095455 (+ 4488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:51.096597 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:51.099014 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:51.099840 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:51.099948 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:51.104323 (+ 4375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:51.104332 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:51.104339 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:51.104392 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:51.108351 (+ 3959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:51.110118 (+ 1767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:51.110131 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:51.110136 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:51.110558 (+ 422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:51.115290 (+ 4732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:51.115297 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:51.115301 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:51.115782 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:51.116069 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:51.119842 (+ 3773us) server_negotiation.cc:299] Negotiation successful
0806 02:48:51.120146 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:48:51.124421 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:51.078089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53812 (local address 127.1.210.126:39673)
0806 02:48:51.078328 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:51.078338 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:51.089381 (+ 11043us) server_negotiation.cc:407] Connection header received
0806 02:48:51.089682 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:51.089691 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:51.089812 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:51.089954 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:51.101353 (+ 11399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:51.102658 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:51.104985 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:51.105706 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:51.105815 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:51.111360 (+ 5545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:51.111371 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:51.111379 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:51.111458 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:51.117433 (+ 5975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:51.118773 (+ 1340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:51.118782 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:51.118789 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:51.119002 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:51.120584 (+ 1582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:51.120618 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:51.120625 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:51.121505 (+ 880us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:51.123558 (+ 2053us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:51.123738 (+ 180us) server_negotiation.cc:299] Negotiation successful
0806 02:48:51.123995 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 02:48:51.127146 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:51.090634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36742 (local address 127.1.210.124:44299)
0806 02:48:51.090855 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:51.090861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:51.099211 (+ 8350us) server_negotiation.cc:407] Connection header received
0806 02:48:51.099890 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:51.099897 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:51.099999 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:51.100075 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:51.105111 (+ 5036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:51.106256 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:51.109685 (+ 3429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:51.110564 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:51.110719 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:51.114260 (+ 3541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:51.114267 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:51.114273 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:51.114322 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:51.119056 (+ 4734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:51.123015 (+ 3959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:51.123022 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:51.123026 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:51.123248 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:51.125061 (+ 1813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:51.125067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:51.125070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:51.125474 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:51.125789 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:51.126446 (+ 657us) server_negotiation.cc:299] Negotiation successful
0806 02:48:51.126742 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
W20260806 02:48:51.139298 15692 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 757edc90130848cd8e9d696d5cc84e0b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:48:51.139567 15692 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:48:51.138795 15584 master_runner.cc:186] Error getting master registration for 127.1.210.125:34911: OK, instance_id { permanent_uuid: "a9912610f343455bab6c9248f5d4a246" instance_seqno: 1785984530421696 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 02:48:51.148439 15584 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 02:48:51.155052 15576 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44299 } attempt: 5
Aug 06 02:48:51 dist-test-slave-8tdl krb5kdc[15420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984530, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.125@KRBTEST.COM for kudu/127.1.210.124@KRBTEST.COM
I20260806 02:48:51.170838 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:51.156311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36744 (local address 127.1.210.124:44299)
0806 02:48:51.156469 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:51.156474 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:51.156488 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:48:51.156532 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:51.156535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:51.156589 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:51.156693 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:51.157489 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:51.158290 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:51.159578 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:51.160169 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:51.160309 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:51.164588 (+ 4279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:51.164594 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:51.164599 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:51.164635 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:51.167746 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:51.168467 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:51.168475 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:51.168481 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:51.168696 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:51.169171 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:51.169176 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:51.169179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:51.169538 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:51.169778 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:51.170177 (+ 399us) server_negotiation.cc:299] Negotiation successful
0806 02:48:51.170404 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:48:51.173077 15580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:51.155688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:44299 (local address 127.0.0.1:36744)
0806 02:48:51.155921 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:51.155939 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:51.156051 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:51.156733 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:51.156737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:51.157180 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:51.157341 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:51.157350 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:51.158440 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:51.158447 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:51.159415 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:51.159430 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:51.160332 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:51.160339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:51.160581 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:51.161243 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:51.161268 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:51.164355 (+ 3087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:51.167897 (+ 3542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:51.167903 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:51.167907 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:51.168336 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:51.168858 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:51.168862 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:51.168865 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:51.169058 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:51.169660 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:51.169666 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:51.169847 (+ 181us) client_negotiation.cc:769] Sending connection context
0806 02:48:51.170022 (+ 175us) client_negotiation.cc:240] Negotiation successful
0806 02:48:51.170371 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260806 02:48:51.178016 15576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9912610f343455bab6c9248f5d4a246: Bootstrap starting.
I20260806 02:48:51.181435 15576 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9912610f343455bab6c9248f5d4a246: Neither blocks nor log segments found. Creating new log.
I20260806 02:48:51.182839 15576 log.cc:824] T 00000000000000000000000000000000 P a9912610f343455bab6c9248f5d4a246: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:51.189666 15576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9912610f343455bab6c9248f5d4a246: No bootstrap required, opened a new log
I20260806 02:48:51.198863 15576 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9912610f343455bab6c9248f5d4a246 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0554fa6958dc4c489f2346d91579428e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39673 } } peers { permanent_uuid: "a9912610f343455bab6c9248f5d4a246" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 34911 } } peers { permanent_uuid: "757edc90130848cd8e9d696d5cc84e0b" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44299 } }
I20260806 02:48:51.199373 15576 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9912610f343455bab6c9248f5d4a246 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:48:51.199529 15576 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9912610f343455bab6c9248f5d4a246 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9912610f343455bab6c9248f5d4a246, State: Initialized, Role: FOLLOWER
I20260806 02:48:51.200284 15576 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9912610f343455bab6c9248f5d4a246 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0554fa6958dc4c489f2346d91579428e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39673 } } peers { permanent_uuid: "a9912610f343455bab6c9248f5d4a246" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 34911 } } peers { permanent_uuid: "757edc90130848cd8e9d696d5cc84e0b" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44299 } }
I20260806 02:48:51.201463 15695 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9912610f343455bab6c9248f5d4a246 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0554fa6958dc4c489f2346d91579428e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39673 } } peers { permanent_uuid: "a9912610f343455bab6c9248f5d4a246" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 34911 } } peers { permanent_uuid: "757edc90130848cd8e9d696d5cc84e0b" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44299 } } }
I20260806 02:48:51.202009 15695 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9912610f343455bab6c9248f5d4a246 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:48:51.202939 15576 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9912610f343455bab6c9248f5d4a246 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 02:48:51.203363 15605 tablet.cc:2369] T 00000000000000000000000000000000 P 757edc90130848cd8e9d696d5cc84e0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:48:51.219640 15508 master_runner.cc:263] No leader master found from master a9912610f343455bab6c9248f5d4a246
I20260806 02:48:51.219858 15508 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260806 02:48:51.230628 15706 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a9912610f343455bab6c9248f5d4a246: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:48:51.230933 15706 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:48:51.274499 15660 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0554fa6958dc4c489f2346d91579428e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:48:51.274936 15660 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0554fa6958dc4c489f2346d91579428e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0554fa6958dc4c489f2346d91579428e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39673 } } peers { permanent_uuid: "a9912610f343455bab6c9248f5d4a246" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 34911 } } peers { permanent_uuid: "757edc90130848cd8e9d696d5cc84e0b" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44299 } }
I20260806 02:48:51.276784 15660 leader_election.cc:288] T 00000000000000000000000000000000 P 0554fa6958dc4c489f2346d91579428e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9912610f343455bab6c9248f5d4a246 (127.1.210.125:34911), 757edc90130848cd8e9d696d5cc84e0b (127.1.210.124:44299)
I20260806 02:48:51.278281 15626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0554fa6958dc4c489f2346d91579428e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "757edc90130848cd8e9d696d5cc84e0b" is_pre_election: true
I20260806 02:48:51.279472 15550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0554fa6958dc4c489f2346d91579428e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9912610f343455bab6c9248f5d4a246" is_pre_election: true
I20260806 02:48:51.280153 15550 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a9912610f343455bab6c9248f5d4a246 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0554fa6958dc4c489f2346d91579428e in term 0.
I20260806 02:48:51.281057 15452 leader_election.cc:302] T 00000000000000000000000000000000 P 0554fa6958dc4c489f2346d91579428e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0554fa6958dc4c489f2346d91579428e, a9912610f343455bab6c9248f5d4a246; no voters:
I20260806 02:48:51.281841 15626 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 757edc90130848cd8e9d696d5cc84e0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0554fa6958dc4c489f2346d91579428e in term 0.
I20260806 02:48:51.286626 15660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0554fa6958dc4c489f2346d91579428e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:48:51.286829 15660 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0554fa6958dc4c489f2346d91579428e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:48:51.286936 15660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0554fa6958dc4c489f2346d91579428e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:48:51.290431 15660 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0554fa6958dc4c489f2346d91579428e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0554fa6958dc4c489f2346d91579428e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39673 } } peers { permanent_uuid: "a9912610f343455bab6c9248f5d4a246" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 34911 } } peers { permanent_uuid: "757edc90130848cd8e9d696d5cc84e0b" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44299 } }
I20260806 02:48:51.292400 15550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0554fa6958dc4c489f2346d91579428e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9912610f343455bab6c9248f5d4a246"
I20260806 02:48:51.292651 15550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9912610f343455bab6c9248f5d4a246 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:48:51.296797 15550 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a9912610f343455bab6c9248f5d4a246 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0554fa6958dc4c489f2346d91579428e in term 1.
I20260806 02:48:51.297402 15452 leader_election.cc:302] T 00000000000000000000000000000000 P 0554fa6958dc4c489f2346d91579428e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0554fa6958dc4c489f2346d91579428e, a9912610f343455bab6c9248f5d4a246; no voters:
I20260806 02:48:51.301327 15626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0554fa6958dc4c489f2346d91579428e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "757edc90130848cd8e9d696d5cc84e0b"
I20260806 02:48:51.301594 15626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 757edc90130848cd8e9d696d5cc84e0b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:48:51.305608 15626 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 757edc90130848cd8e9d696d5cc84e0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0554fa6958dc4c489f2346d91579428e in term 1.
I20260806 02:48:51.314931 15660 leader_election.cc:288] T 00000000000000000000000000000000 P 0554fa6958dc4c489f2346d91579428e [CANDIDATE]: Term 1 election: Requested vote from peers a9912610f343455bab6c9248f5d4a246 (127.1.210.125:34911), 757edc90130848cd8e9d696d5cc84e0b (127.1.210.124:44299)
I20260806 02:48:51.315511 15660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0554fa6958dc4c489f2346d91579428e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:48:51.316010 15660 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0554fa6958dc4c489f2346d91579428e [term 1 LEADER]: Becoming Leader. State: Replica: 0554fa6958dc4c489f2346d91579428e, State: Running, Role: LEADER
I20260806 02:48:51.316970 15660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0554fa6958dc4c489f2346d91579428e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0554fa6958dc4c489f2346d91579428e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39673 } } peers { permanent_uuid: "a9912610f343455bab6c9248f5d4a246" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 34911 } } peers { permanent_uuid: "757edc90130848cd8e9d696d5cc84e0b" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44299 } }
I20260806 02:48:51.339322 15708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0554fa6958dc4c489f2346d91579428e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0554fa6958dc4c489f2346d91579428e. Latest consensus state: current_term: 1 leader_uuid: "0554fa6958dc4c489f2346d91579428e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0554fa6958dc4c489f2346d91579428e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39673 } } peers { permanent_uuid: "a9912610f343455bab6c9248f5d4a246" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 34911 } } peers { permanent_uuid: "757edc90130848cd8e9d696d5cc84e0b" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44299 } } }
I20260806 02:48:51.339910 15708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0554fa6958dc4c489f2346d91579428e [sys.catalog]: This master's current role is: LEADER
I20260806 02:48:51.347294 15710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:48:51.357789 15710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:48:51.388643 15626 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 757edc90130848cd8e9d696d5cc84e0b [term 1 FOLLOWER]: Refusing update from remote peer 0554fa6958dc4c489f2346d91579428e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:48:51.389846 15708 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0554fa6958dc4c489f2346d91579428e [LEADER]: Connected to new peer: Peer: permanent_uuid: "757edc90130848cd8e9d696d5cc84e0b" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:48:51.397396 15550 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a9912610f343455bab6c9248f5d4a246 [term 1 FOLLOWER]: Refusing update from remote peer 0554fa6958dc4c489f2346d91579428e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:48:51.398449 15708 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0554fa6958dc4c489f2346d91579428e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9912610f343455bab6c9248f5d4a246" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 34911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 02:48:51.410257 15529 tablet.cc:2369] T 00000000000000000000000000000000 P a9912610f343455bab6c9248f5d4a246: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:48:51.433336 15695 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9912610f343455bab6c9248f5d4a246 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0554fa6958dc4c489f2346d91579428e. Latest consensus state: current_term: 1 leader_uuid: "0554fa6958dc4c489f2346d91579428e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0554fa6958dc4c489f2346d91579428e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39673 } } peers { permanent_uuid: "a9912610f343455bab6c9248f5d4a246" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 34911 } } peers { permanent_uuid: "757edc90130848cd8e9d696d5cc84e0b" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44299 } } }
I20260806 02:48:51.433651 15695 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9912610f343455bab6c9248f5d4a246 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:48:51.438723 15711 mvcc.cc:205] Tried to move back new op lower bound from 7315392640554491904 to 7315392640283963392. Current Snapshot: MvccSnapshot[applied={T|T < 7315392640554491904}]
I20260806 02:48:51.443413 15713 mvcc.cc:205] Tried to move back new op lower bound from 7315392640554491904 to 7315392640283963392. Current Snapshot: MvccSnapshot[applied={T|T < 7315392640554491904}]
I20260806 02:48:51.447154 15708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0554fa6958dc4c489f2346d91579428e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0554fa6958dc4c489f2346d91579428e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0554fa6958dc4c489f2346d91579428e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39673 } } peers { permanent_uuid: "a9912610f343455bab6c9248f5d4a246" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 34911 } } peers { permanent_uuid: "757edc90130848cd8e9d696d5cc84e0b" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44299 } } }
I20260806 02:48:51.447484 15708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0554fa6958dc4c489f2346d91579428e [sys.catalog]: This master's current role is: LEADER
I20260806 02:48:51.449653 15710 catalog_manager.cc:1347] Generated new cluster ID: 2a9744fe331c46a486c8152e6a1cd0af
I20260806 02:48:51.449792 15710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:48:51.452684 15695 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9912610f343455bab6c9248f5d4a246 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0554fa6958dc4c489f2346d91579428e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0554fa6958dc4c489f2346d91579428e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39673 } } peers { permanent_uuid: "a9912610f343455bab6c9248f5d4a246" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 34911 } } peers { permanent_uuid: "757edc90130848cd8e9d696d5cc84e0b" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44299 } } }
I20260806 02:48:51.452977 15695 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9912610f343455bab6c9248f5d4a246 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:48:51.457717 15680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 757edc90130848cd8e9d696d5cc84e0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0554fa6958dc4c489f2346d91579428e. Latest consensus state: current_term: 1 leader_uuid: "0554fa6958dc4c489f2346d91579428e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0554fa6958dc4c489f2346d91579428e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39673 } } peers { permanent_uuid: "a9912610f343455bab6c9248f5d4a246" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 34911 } } peers { permanent_uuid: "757edc90130848cd8e9d696d5cc84e0b" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44299 } } }
I20260806 02:48:51.458052 15680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 757edc90130848cd8e9d696d5cc84e0b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:48:51.461009 15660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0554fa6958dc4c489f2346d91579428e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0554fa6958dc4c489f2346d91579428e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0554fa6958dc4c489f2346d91579428e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39673 } } peers { permanent_uuid: "a9912610f343455bab6c9248f5d4a246" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 34911 } } peers { permanent_uuid: "757edc90130848cd8e9d696d5cc84e0b" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44299 } } }
I20260806 02:48:51.461880 15660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0554fa6958dc4c489f2346d91579428e [sys.catalog]: This master's current role is: LEADER
I20260806 02:48:51.465813 15712 mvcc.cc:205] Tried to move back new op lower bound from 7315392640554491904 to 7315392640283963392. Current Snapshot: MvccSnapshot[applied={T|T < 7315392640554491904}]
I20260806 02:48:51.471081 15680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 757edc90130848cd8e9d696d5cc84e0b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0554fa6958dc4c489f2346d91579428e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0554fa6958dc4c489f2346d91579428e" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39673 } } peers { permanent_uuid: "a9912610f343455bab6c9248f5d4a246" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 34911 } } peers { permanent_uuid: "757edc90130848cd8e9d696d5cc84e0b" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 44299 } } }
I20260806 02:48:51.471364 15680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 757edc90130848cd8e9d696d5cc84e0b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:48:51.501754 15710 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:48:51.503744 15710 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:48:51.535413 15710 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0554fa6958dc4c489f2346d91579428e: Generated new TSK 0
I20260806 02:48:51.536644 15710 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:48:52.141739 15692 catalog_manager.cc:1259] Loaded cluster ID: 2a9744fe331c46a486c8152e6a1cd0af
I20260806 02:48:52.141856 15692 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 757edc90130848cd8e9d696d5cc84e0b: loading cluster ID for follower catalog manager: success
I20260806 02:48:52.144676 15692 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 757edc90130848cd8e9d696d5cc84e0b: acquiring CA information for follower catalog manager: success
I20260806 02:48:52.146761 15692 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 757edc90130848cd8e9d696d5cc84e0b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:48:52.233451 15706 catalog_manager.cc:1259] Loaded cluster ID: 2a9744fe331c46a486c8152e6a1cd0af
I20260806 02:48:52.233575 15706 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a9912610f343455bab6c9248f5d4a246: loading cluster ID for follower catalog manager: success
I20260806 02:48:52.236524 15706 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a9912610f343455bab6c9248f5d4a246: acquiring CA information for follower catalog manager: success
I20260806 02:48:52.238783 15706 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a9912610f343455bab6c9248f5d4a246: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:48:52.707659 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 15433
I20260806 02:48:52.708976 15496 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:48:52.823081 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 15433
I20260806 02:48:52.848390 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 15508
I20260806 02:48:52.849733 15570 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:48:52.959789 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 15508
I20260806 02:48:52.983873 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 15584
I20260806 02:48:52.984990 15647 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:48:53.095697 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 15584
2026-08-06T02:48:53Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5190 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:48:53 dist-test-slave-8tdl krb5kdc[15731](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:48:53 dist-test-slave-8tdl krb5kdc[15731](info): set up 2 sockets
Aug 06 02:48:53 dist-test-slave-8tdl krb5kdc[15731](info): commencing operation
krb5kdc: starting...
W20260806 02:48:54.238844 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:48:54 dist-test-slave-8tdl krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:48:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:48:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
I20260806 02:48:54.527849 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:39521
--webserver_interface=127.1.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:36599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:39521
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:48:54.882995 15744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:54.883306 15744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:54.883479 15744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:54.890798 15744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:48:54.890920 15744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:54.891011 15744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:54.891136 15744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:48:54.891193 15744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:48:54.901418 15744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:36599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:39521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:39521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:48:54.902868 15744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:54.904292 15744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:54.916955 15749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:54.917032 15750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:54.916996 15752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:54.918107 15744 server_base.cc:1034] running on GCE node
I20260806 02:48:54.919044 15744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:54.920647 15744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:54.921892 15744 hybrid_clock.cc:648] HybridClock initialized: now 1785984534921820 us; error 63 us; skew 500 ppm
I20260806 02:48:54.922279 15744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:48:54 dist-test-slave-8tdl krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984534, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:54.929561 15744 init.cc:377] Logged in from keytab as kudu/127.1.210.126@KRBTEST.COM (short username kudu)
I20260806 02:48:54.931576 15744 webserver.cc:460] Webserver started at http://127.1.210.126:33201/ using document root <none> and password file <none>
I20260806 02:48:54.932415 15744 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:54.932601 15744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:54.932940 15744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:48:54.936436 15744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "735334e6dcc24f719cb85709716cf95c"
format_stamp: "Formatted at 2026-08-06 02:48:54 on dist-test-slave-8tdl"
server_key: "d4dfdf2e45b476bf22c97f3309e2db8c"
server_key_iv: "9e8087b28c39a3c4229ed1b76231a485"
server_key_version: "encryptionkey@0"
I20260806 02:48:54.937379 15744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "735334e6dcc24f719cb85709716cf95c"
format_stamp: "Formatted at 2026-08-06 02:48:54 on dist-test-slave-8tdl"
server_key: "d4dfdf2e45b476bf22c97f3309e2db8c"
server_key_iv: "9e8087b28c39a3c4229ed1b76231a485"
server_key_version: "encryptionkey@0"
I20260806 02:48:54.943742 15744 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260806 02:48:54.948112 15759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:54.949836 15744 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:48:54.950071 15744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "735334e6dcc24f719cb85709716cf95c"
format_stamp: "Formatted at 2026-08-06 02:48:54 on dist-test-slave-8tdl"
server_key: "d4dfdf2e45b476bf22c97f3309e2db8c"
server_key_iv: "9e8087b28c39a3c4229ed1b76231a485"
server_key_version: "encryptionkey@0"
I20260806 02:48:54.950309 15744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:54.966440 15744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:54.971333 15744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:54.971674 15744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:54.992835 15744 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:39521
I20260806 02:48:54.992858 15811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:39521 every 8 connection(s)
I20260806 02:48:54.994378 15744 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:48:54.996217 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 15744
I20260806 02:48:54.996479 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:48:54.997377 1865 external_mini_cluster.cc:1428] Setting key fef5f5046f9e5c9508e3551923c8f1a6
I20260806 02:48:54.999686 15812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 02:48:55 dist-test-slave-8tdl krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:48:55.015483 15812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 735334e6dcc24f719cb85709716cf95c: Bootstrap starting.
I20260806 02:48:55.020069 15812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 735334e6dcc24f719cb85709716cf95c: Neither blocks nor log segments found. Creating new log.
I20260806 02:48:55.021238 15812 log.cc:824] T 00000000000000000000000000000000 P 735334e6dcc24f719cb85709716cf95c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:55.024637 15812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 735334e6dcc24f719cb85709716cf95c: No bootstrap required, opened a new log
I20260806 02:48:55.025321 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:54.999479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47258 (local address 127.1.210.126:39521)
0806 02:48:55.000405 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:55.000418 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:55.000462 (+ 44us) server_negotiation.cc:407] Connection header received
0806 02:48:55.002295 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:55.002323 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:55.002884 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:55.003309 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:55.005510 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.007437 (+ 1927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.009483 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.010294 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.010464 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:55.016160 (+ 5696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:55.016171 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:55.016188 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:55.016239 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:55.020398 (+ 4159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:55.021344 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:55.021354 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:55.021361 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:55.021633 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:55.022110 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:55.022119 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:55.022122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:55.022820 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:55.023133 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:55.023663 (+ 530us) server_negotiation.cc:299] Negotiation successful
0806 02:48:55.024134 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":444,"threads_started":1}
I20260806 02:48:55.029646 1865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:48:55.033913 15812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 735334e6dcc24f719cb85709716cf95c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "735334e6dcc24f719cb85709716cf95c" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39521 } }
I20260806 02:48:55.034340 15812 raft_consensus.cc:374] T 00000000000000000000000000000000 P 735334e6dcc24f719cb85709716cf95c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:48:55.034456 15812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 735334e6dcc24f719cb85709716cf95c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 735334e6dcc24f719cb85709716cf95c, State: Initialized, Role: FOLLOWER
I20260806 02:48:55.035233 15812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 735334e6dcc24f719cb85709716cf95c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "735334e6dcc24f719cb85709716cf95c" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39521 } }
I20260806 02:48:55.035496 15812 raft_consensus.cc:388] T 00000000000000000000000000000000 P 735334e6dcc24f719cb85709716cf95c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:48:55.035650 15812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 735334e6dcc24f719cb85709716cf95c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:48:55.035832 15812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 735334e6dcc24f719cb85709716cf95c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:48:55.038148 15812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 735334e6dcc24f719cb85709716cf95c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "735334e6dcc24f719cb85709716cf95c" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39521 } }
I20260806 02:48:55.038770 15812 leader_election.cc:302] T 00000000000000000000000000000000 P 735334e6dcc24f719cb85709716cf95c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 735334e6dcc24f719cb85709716cf95c; no voters:
I20260806 02:48:55.039433 15812 leader_election.cc:288] T 00000000000000000000000000000000 P 735334e6dcc24f719cb85709716cf95c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:48:55.039621 15817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 735334e6dcc24f719cb85709716cf95c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:48:55.040377 15817 raft_consensus.cc:686] T 00000000000000000000000000000000 P 735334e6dcc24f719cb85709716cf95c [term 1 LEADER]: Becoming Leader. State: Replica: 735334e6dcc24f719cb85709716cf95c, State: Running, Role: LEADER
I20260806 02:48:55.040994 15817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 735334e6dcc24f719cb85709716cf95c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "735334e6dcc24f719cb85709716cf95c" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39521 } }
I20260806 02:48:55.041952 15812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 735334e6dcc24f719cb85709716cf95c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:48:55.044159 15818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 735334e6dcc24f719cb85709716cf95c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "735334e6dcc24f719cb85709716cf95c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "735334e6dcc24f719cb85709716cf95c" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39521 } } }
I20260806 02:48:55.044162 15819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 735334e6dcc24f719cb85709716cf95c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 735334e6dcc24f719cb85709716cf95c. Latest consensus state: current_term: 1 leader_uuid: "735334e6dcc24f719cb85709716cf95c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "735334e6dcc24f719cb85709716cf95c" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 39521 } } }
I20260806 02:48:55.044654 15818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 735334e6dcc24f719cb85709716cf95c [sys.catalog]: This master's current role is: LEADER
I20260806 02:48:55.044718 15819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 735334e6dcc24f719cb85709716cf95c [sys.catalog]: This master's current role is: LEADER
I20260806 02:48:55.047091 15823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:48:55.052815 15823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:48:55.063356 15823 catalog_manager.cc:1347] Generated new cluster ID: 5ab8b5f220ef4d239f843cdd1f6ac0ce
I20260806 02:48:55.063584 15823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:48:55.100287 15823 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:48:55.101840 15823 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:48:55.130615 15823 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 735334e6dcc24f719cb85709716cf95c: Generated new TSK 0
I20260806 02:48:55.131876 15823 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:48:55.143636 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 15744
I20260806 02:48:55.145118 15807 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:48:55.245751 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 15744
2026-08-06T02:48:55Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2146 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:48:55 dist-test-slave-8tdl krb5kdc[15840](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:48:55 dist-test-slave-8tdl krb5kdc[15840](info): set up 2 sockets
Aug 06 02:48:55 dist-test-slave-8tdl krb5kdc[15840](info): commencing operation
krb5kdc: starting...
W20260806 02:48:56.394601 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.001s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:48:56 dist-test-slave-8tdl krb5kdc[15840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:48:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:48:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
I20260806 02:48:56.678665 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:45859
--webserver_interface=127.1.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:45859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:48:57.030548 15854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:57.030985 15854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:57.031095 15854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:57.038931 15854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:48:57.039065 15854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:57.039121 15854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:57.039181 15854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:48:57.039220 15854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:48:57.049832 15854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:45859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:45859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:48:57.051388 15854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:57.052764 15854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:57.065675 15859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:57.065647 15860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:57.065909 15862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:57.066557 15854 server_base.cc:1034] running on GCE node
I20260806 02:48:57.067456 15854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:57.069020 15854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:57.070242 15854 hybrid_clock.cc:648] HybridClock initialized: now 1785984537070172 us; error 61 us; skew 500 ppm
I20260806 02:48:57.070623 15854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:48:57 dist-test-slave-8tdl krb5kdc[15840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984537, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:57.074265 15854 init.cc:377] Logged in from keytab as kudu/127.1.210.126@KRBTEST.COM (short username kudu)
I20260806 02:48:57.076113 15854 webserver.cc:460] Webserver started at http://127.1.210.126:35143/ using document root <none> and password file <none>
I20260806 02:48:57.076871 15854 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:57.077138 15854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:57.077533 15854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:48:57.080886 15854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "1b645c239c754d49a17f96a216c4af80"
format_stamp: "Formatted at 2026-08-06 02:48:57 on dist-test-slave-8tdl"
server_key: "80246ef024936cd5ac4eb7aed35dc814"
server_key_iv: "267a2538434db8f71d260a2b8d04f70e"
server_key_version: "encryptionkey@0"
I20260806 02:48:57.081728 15854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "1b645c239c754d49a17f96a216c4af80"
format_stamp: "Formatted at 2026-08-06 02:48:57 on dist-test-slave-8tdl"
server_key: "80246ef024936cd5ac4eb7aed35dc814"
server_key_iv: "267a2538434db8f71d260a2b8d04f70e"
server_key_version: "encryptionkey@0"
I20260806 02:48:57.087500 15854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260806 02:48:57.091574 15869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:57.093551 15854 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:48:57.093789 15854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "1b645c239c754d49a17f96a216c4af80"
format_stamp: "Formatted at 2026-08-06 02:48:57 on dist-test-slave-8tdl"
server_key: "80246ef024936cd5ac4eb7aed35dc814"
server_key_iv: "267a2538434db8f71d260a2b8d04f70e"
server_key_version: "encryptionkey@0"
I20260806 02:48:57.094033 15854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:57.107240 15854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:57.112349 15854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:57.112682 15854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:57.134402 15854 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:45859
I20260806 02:48:57.134441 15921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:45859 every 8 connection(s)
I20260806 02:48:57.136077 15854 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:48:57.137590 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 15854
I20260806 02:48:57.137833 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:48:57.138787 1865 external_mini_cluster.cc:1428] Setting key aa0e44da0eb946ff86649d84f977e23e
I20260806 02:48:57.141606 15922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 02:48:57 dist-test-slave-8tdl krb5kdc[15840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:48:57.157232 15922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80: Bootstrap starting.
I20260806 02:48:57.161056 15922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80: Neither blocks nor log segments found. Creating new log.
I20260806 02:48:57.162223 15922 log.cc:824] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:57.165786 15922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80: No bootstrap required, opened a new log
I20260806 02:48:57.165825 15925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:57.141408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35660 (local address 127.1.210.126:45859)
0806 02:48:57.142338 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:57.142351 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:57.142399 (+ 48us) server_negotiation.cc:407] Connection header received
0806 02:48:57.144219 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:57.144244 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:57.144690 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:57.145066 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:57.147152 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:57.148630 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:57.150573 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:57.151651 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:57.151812 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:57.157481 (+ 5669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:57.157492 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:57.157516 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:57.157562 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:57.161075 (+ 3513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:57.161923 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:57.161935 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:57.161943 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:57.162137 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:57.162604 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:57.162609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:57.162616 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:57.163545 (+ 929us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:57.163862 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:57.164333 (+ 471us) server_negotiation.cc:299] Negotiation successful
0806 02:48:57.164756 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":450,"threads_started":1}
I20260806 02:48:57.174499 15922 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b645c239c754d49a17f96a216c4af80" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45859 } }
I20260806 02:48:57.175005 15922 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:48:57.175122 15922 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b645c239c754d49a17f96a216c4af80, State: Initialized, Role: FOLLOWER
I20260806 02:48:57.175706 15922 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b645c239c754d49a17f96a216c4af80" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45859 } }
I20260806 02:48:57.175958 15922 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:48:57.176113 15922 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:48:57.176293 15922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:48:57.178977 15922 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b645c239c754d49a17f96a216c4af80" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45859 } }
I20260806 02:48:57.179514 15922 leader_election.cc:302] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b645c239c754d49a17f96a216c4af80; no voters:
I20260806 02:48:57.180172 15922 leader_election.cc:288] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:48:57.180305 15927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:48:57.181114 15927 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [term 1 LEADER]: Becoming Leader. State: Replica: 1b645c239c754d49a17f96a216c4af80, State: Running, Role: LEADER
I20260806 02:48:57.181753 15927 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b645c239c754d49a17f96a216c4af80" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45859 } }
I20260806 02:48:57.182435 15922 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:48:57.184500 15929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b645c239c754d49a17f96a216c4af80. Latest consensus state: current_term: 1 leader_uuid: "1b645c239c754d49a17f96a216c4af80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b645c239c754d49a17f96a216c4af80" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45859 } } }
I20260806 02:48:57.184923 15929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [sys.catalog]: This master's current role is: LEADER
I20260806 02:48:57.185129 15928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b645c239c754d49a17f96a216c4af80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b645c239c754d49a17f96a216c4af80" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45859 } } }
I20260806 02:48:57.185660 15928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [sys.catalog]: This master's current role is: LEADER
I20260806 02:48:57.194993 15932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:48:57.203349 15932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:48:57.217129 15932 catalog_manager.cc:1347] Generated new cluster ID: 67ed54e8ea694172866975565c9c0c1b
I20260806 02:48:57.217262 15932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:48:57.230487 15932 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:48:57.231950 15932 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:48:57.240419 15932 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80: Generated new TSK 0
I20260806 02:48:57.241410 15932 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab.
Entry for principal kudu/127.1.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab.
I20260806 02:48:57.327561 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.65:0
--local_ip_for_outbound_sockets=127.1.210.65
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:45859
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:48:57.725978 15949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:57.726483 15949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:57.726658 15949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:57.739529 15949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:57.739708 15949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:57.739992 15949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.65
I20260806 02:48:57.754820 15949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:45859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.65
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:48:57.756491 15949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:57.758181 15949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:57.773003 15954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:57.772945 15957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:57.773028 15955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:57.773799 15949 server_base.cc:1034] running on GCE node
I20260806 02:48:57.774485 15949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:57.775621 15949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:57.776851 15949 hybrid_clock.cc:648] HybridClock initialized: now 1785984537776777 us; error 65 us; skew 500 ppm
I20260806 02:48:57.777304 15949 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:48:57 dist-test-slave-8tdl krb5kdc[15840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984537, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:57.780998 15949 init.cc:377] Logged in from keytab as kudu/127.1.210.65@KRBTEST.COM (short username kudu)
I20260806 02:48:57.782670 15949 webserver.cc:460] Webserver started at http://127.1.210.65:44417/ using document root <none> and password file <none>
I20260806 02:48:57.783600 15949 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:57.783735 15949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:57.784075 15949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:48:57.787339 15949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/data/instance:
uuid: "a2a25e0aaf014da8ab4bac97925b34d5"
format_stamp: "Formatted at 2026-08-06 02:48:57 on dist-test-slave-8tdl"
server_key: "8e9a6c4da6256d24c0a019d170b7fbe8"
server_key_iv: "d9f8cfd24fdf0d01605fb2199f920d70"
server_key_version: "encryptionkey@0"
I20260806 02:48:57.788203 15949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance:
uuid: "a2a25e0aaf014da8ab4bac97925b34d5"
format_stamp: "Formatted at 2026-08-06 02:48:57 on dist-test-slave-8tdl"
server_key: "8e9a6c4da6256d24c0a019d170b7fbe8"
server_key_iv: "d9f8cfd24fdf0d01605fb2199f920d70"
server_key_version: "encryptionkey@0"
I20260806 02:48:57.794076 15949 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 02:48:57.798599 15964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:57.800376 15949 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:48:57.800571 15949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/wal
uuid: "a2a25e0aaf014da8ab4bac97925b34d5"
format_stamp: "Formatted at 2026-08-06 02:48:57 on dist-test-slave-8tdl"
server_key: "8e9a6c4da6256d24c0a019d170b7fbe8"
server_key_iv: "d9f8cfd24fdf0d01605fb2199f920d70"
server_key_version: "encryptionkey@0"
I20260806 02:48:57.800840 15949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:57.820360 15949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:57.825093 15949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:57.825362 15949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:57.826328 15949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:48:57.827793 15949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:48:57.827868 15949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:57.827956 15949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:48:57.828001 15949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:57.864236 15949 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.65:35323
I20260806 02:48:57.864285 16078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.65:35323 every 8 connection(s)
I20260806 02:48:57.865612 15949 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
I20260806 02:48:57.871805 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 15949
I20260806 02:48:57.872110 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance
I20260806 02:48:57.873023 1865 external_mini_cluster.cc:1428] Setting key a4b046678c0f470eea8a33fb5a9dd1c2
Aug 06 02:48:57 dist-test-slave-8tdl krb5kdc[15840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984537, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:48:57.889694 16081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:57.868350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:58705 (local address 127.1.210.126:45859)
0806 02:48:57.868872 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:57.868876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:57.870003 (+ 1127us) server_negotiation.cc:407] Connection header received
0806 02:48:57.872007 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:57.872013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:57.872147 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:57.872278 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:57.874489 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:57.875573 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:57.878272 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:57.878666 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:57.878829 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:57.883554 (+ 4725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:57.883559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:57.883562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:57.883601 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:57.886485 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:57.887269 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:57.887277 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:57.887282 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:57.887412 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:57.887850 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:57.887855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:57.887863 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:57.888159 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:57.888316 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:57.889036 (+ 720us) server_negotiation.cc:299] Negotiation successful
0806 02:48:57.889291 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":284,"threads_started":1}
I20260806 02:48:57.890777 16082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:57.869057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.65:58705)
0806 02:48:57.869821 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:57.869867 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:57.871717 (+ 1850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:57.872625 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:57.872638 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:57.873496 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:57.874304 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:57.874326 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:57.875764 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:57.875770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:57.878094 (+ 2324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:57.878116 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:57.878979 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:57.878990 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:57.879344 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:57.880137 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:57.880180 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:57.883292 (+ 3112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:57.886631 (+ 3339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:57.886640 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:57.886658 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:57.887148 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:57.887521 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:57.887526 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:57.887530 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:57.887730 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:57.888288 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:57.888296 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:57.888665 (+ 369us) client_negotiation.cc:769] Sending connection context
0806 02:48:57.888936 (+ 271us) client_negotiation.cc:240] Negotiation successful
0806 02:48:57.889505 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":293,"threads_started":1}
I20260806 02:48:57.892484 16079 heartbeater.cc:347] Connected to a master server at 127.1.210.126:45859
I20260806 02:48:57.892793 16079 heartbeater.cc:464] Registering TS with master...
I20260806 02:48:57.893567 16079 heartbeater.cc:511] Master 127.1.210.126:45859 requested a full tablet report, sending...
I20260806 02:48:57.896150 15887 ts_manager.cc:194] Registered new tserver with Master: a2a25e0aaf014da8ab4bac97925b34d5 (127.1.210.65:35323)
I20260806 02:48:57.898540 15887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.65@KRBTEST.COM'} at 127.1.210.65:58705
WARNING: no policy specified for kudu/127.1.210.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab.
Entry for principal kudu/127.1.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab.
I20260806 02:48:57.938544 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.66:0
--local_ip_for_outbound_sockets=127.1.210.66
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:45859
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:48:58.307732 16085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:58.308029 16085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:58.308130 16085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:58.315485 16085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:58.315600 16085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:58.315713 16085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.66
I20260806 02:48:58.325635 16085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:45859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.66
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:48:58.326992 16085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:58.328311 16085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:58.341972 16090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:58.341938 16091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:58.341995 16093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:58.343130 16085 server_base.cc:1034] running on GCE node
I20260806 02:48:58.343806 16085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:58.344801 16085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:58.346078 16085 hybrid_clock.cc:648] HybridClock initialized: now 1785984538346027 us; error 86 us; skew 500 ppm
I20260806 02:48:58.346444 16085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:48:58 dist-test-slave-8tdl krb5kdc[15840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984538, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:58.349977 16085 init.cc:377] Logged in from keytab as kudu/127.1.210.66@KRBTEST.COM (short username kudu)
I20260806 02:48:58.351536 16085 webserver.cc:460] Webserver started at http://127.1.210.66:34855/ using document root <none> and password file <none>
I20260806 02:48:58.352265 16085 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:58.352380 16085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:58.352667 16085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:48:58.355998 16085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/data/instance:
uuid: "88480c29dea44647ba7cc9e90c927baa"
format_stamp: "Formatted at 2026-08-06 02:48:58 on dist-test-slave-8tdl"
server_key: "c296c893bba7c99aa5b7ed43e32966e4"
server_key_iv: "ba3458ecfafe0798a0e8692548627204"
server_key_version: "encryptionkey@0"
I20260806 02:48:58.356819 16085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance:
uuid: "88480c29dea44647ba7cc9e90c927baa"
format_stamp: "Formatted at 2026-08-06 02:48:58 on dist-test-slave-8tdl"
server_key: "c296c893bba7c99aa5b7ed43e32966e4"
server_key_iv: "ba3458ecfafe0798a0e8692548627204"
server_key_version: "encryptionkey@0"
I20260806 02:48:58.362582 16085 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 02:48:58.366469 16100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:58.368132 16085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:48:58.368284 16085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/wal
uuid: "88480c29dea44647ba7cc9e90c927baa"
format_stamp: "Formatted at 2026-08-06 02:48:58 on dist-test-slave-8tdl"
server_key: "c296c893bba7c99aa5b7ed43e32966e4"
server_key_iv: "ba3458ecfafe0798a0e8692548627204"
server_key_version: "encryptionkey@0"
I20260806 02:48:58.368465 16085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:58.391173 16085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:58.396322 16085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:58.396610 16085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:58.397723 16085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:48:58.399514 16085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:48:58.399601 16085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:58.399706 16085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:48:58.399755 16085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:58.436125 16085 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.66:45613
I20260806 02:48:58.436226 16214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.66:45613 every 8 connection(s)
I20260806 02:48:58.437529 16085 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
I20260806 02:48:58.447367 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 16085
I20260806 02:48:58.447574 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance
I20260806 02:48:58.448026 1865 external_mini_cluster.cc:1428] Setting key e8bce2b9918de3b08f9dc769c9034cce
Aug 06 02:48:58 dist-test-slave-8tdl krb5kdc[15840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984538, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:48:58.459378 16217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:58.440446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:49453 (local address 127.1.210.126:45859)
0806 02:48:58.441012 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:58.441016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:58.441944 (+ 928us) server_negotiation.cc:407] Connection header received
0806 02:48:58.444024 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:58.444028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:58.444101 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:58.444193 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:58.445742 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:58.446491 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:58.448362 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:58.448895 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:58.449048 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:58.453220 (+ 4172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:58.453225 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:58.453229 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:58.453263 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:58.456051 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:58.456863 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:58.456868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:58.456871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:58.457024 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:58.457487 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:58.457496 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:58.457498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:58.457773 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:58.457937 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:58.458731 (+ 794us) server_negotiation.cc:299] Negotiation successful
0806 02:48:58.458996 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":263,"threads_started":1}
I20260806 02:48:58.460433 16218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:58.441055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.66:49453)
0806 02:48:58.441762 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:58.441816 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:58.443711 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:58.444415 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:58.444425 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:58.444993 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:58.445567 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:58.445585 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:58.446655 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:58.446659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:58.448209 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:58.448221 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:58.449137 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:58.449145 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:58.449404 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:58.450127 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:58.450184 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:58.452965 (+ 2781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:58.456217 (+ 3252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:58.456227 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:58.456246 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:58.456716 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:58.457144 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:58.457149 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:58.457153 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:58.457361 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:58.457909 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:58.457917 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:58.458298 (+ 381us) client_negotiation.cc:769] Sending connection context
0806 02:48:58.458588 (+ 290us) client_negotiation.cc:240] Negotiation successful
0806 02:48:58.459199 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":263,"threads_started":1}
I20260806 02:48:58.462041 16215 heartbeater.cc:347] Connected to a master server at 127.1.210.126:45859
I20260806 02:48:58.462397 16215 heartbeater.cc:464] Registering TS with master...
I20260806 02:48:58.463420 16215 heartbeater.cc:511] Master 127.1.210.126:45859 requested a full tablet report, sending...
I20260806 02:48:58.465214 15887 ts_manager.cc:194] Registered new tserver with Master: 88480c29dea44647ba7cc9e90c927baa (127.1.210.66:45613)
I20260806 02:48:58.466557 15887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.66@KRBTEST.COM'} at 127.1.210.66:49453
WARNING: no policy specified for kudu/127.1.210.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab.
Entry for principal kudu/127.1.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab.
I20260806 02:48:58.513840 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.67:0
--local_ip_for_outbound_sockets=127.1.210.67
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:45859
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:48:58.856217 16221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:58.856534 16221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:58.856632 16221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:58.864717 16221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:58.864876 16221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:58.865113 16221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.67
I20260806 02:48:58.877161 16221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:45859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.67
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:48:58.878564 16221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:58.879992 16221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:58.893797 16226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:58.893880 16227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:58.894320 16229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:58.894697 16221 server_base.cc:1034] running on GCE node
I20260806 02:48:58.895406 16221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:58.896441 16221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:58.897686 16221 hybrid_clock.cc:648] HybridClock initialized: now 1785984538897612 us; error 67 us; skew 500 ppm
I20260806 02:48:58.898031 16221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:48:58 dist-test-slave-8tdl krb5kdc[15840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984538, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:58.901566 16221 init.cc:377] Logged in from keytab as kudu/127.1.210.67@KRBTEST.COM (short username kudu)
I20260806 02:48:58.901808 16079 heartbeater.cc:503] Master 127.1.210.126:45859 was elected leader, sending a full tablet report...
I20260806 02:48:58.903563 16221 webserver.cc:460] Webserver started at http://127.1.210.67:38321/ using document root <none> and password file <none>
I20260806 02:48:58.904510 16221 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:58.904659 16221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:58.904985 16221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:48:58.908370 16221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/data/instance:
uuid: "f24ecb82f42249ee97edfab14084c27c"
format_stamp: "Formatted at 2026-08-06 02:48:58 on dist-test-slave-8tdl"
server_key: "d7cd5a9d507b9dda6592ef516e507b05"
server_key_iv: "4b65943704d813758b31e6d394e0a541"
server_key_version: "encryptionkey@0"
I20260806 02:48:58.909263 16221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance:
uuid: "f24ecb82f42249ee97edfab14084c27c"
format_stamp: "Formatted at 2026-08-06 02:48:58 on dist-test-slave-8tdl"
server_key: "d7cd5a9d507b9dda6592ef516e507b05"
server_key_iv: "4b65943704d813758b31e6d394e0a541"
server_key_version: "encryptionkey@0"
I20260806 02:48:58.915248 16221 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 02:48:58.919282 16236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:58.920913 16221 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:48:58.921123 16221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/wal
uuid: "f24ecb82f42249ee97edfab14084c27c"
format_stamp: "Formatted at 2026-08-06 02:48:58 on dist-test-slave-8tdl"
server_key: "d7cd5a9d507b9dda6592ef516e507b05"
server_key_iv: "4b65943704d813758b31e6d394e0a541"
server_key_version: "encryptionkey@0"
I20260806 02:48:58.921332 16221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:58.943483 16221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:58.948484 16221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:58.948915 16221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:58.950040 16221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:48:58.951665 16221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:48:58.951743 16221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:58.951880 16221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:48:58.951928 16221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:58.987891 16221 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.67:44225
I20260806 02:48:58.987954 16348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.67:44225 every 8 connection(s)
I20260806 02:48:58.989292 16221 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
I20260806 02:48:58.992713 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 16221
I20260806 02:48:58.992909 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance
I20260806 02:48:58.993547 1865 external_mini_cluster.cc:1428] Setting key fde770b77a51b7f04fb8c57b447a512f
Aug 06 02:48:58 dist-test-slave-8tdl krb5kdc[15840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984538, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:48:59.012465 16351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:58.992086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:56503 (local address 127.1.210.126:45859)
0806 02:48:58.992604 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:58.992610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:58.994168 (+ 1558us) server_negotiation.cc:407] Connection header received
0806 02:48:58.996214 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:58.996220 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:58.996306 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:58.996431 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:58.998454 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:58.999393 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:59.001145 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:59.001587 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:59.001715 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:59.005770 (+ 4055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:59.005775 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:59.005779 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:59.005818 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:59.008942 (+ 3124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:59.009792 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:59.009800 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:59.009803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:59.009946 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:59.010418 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:59.010426 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:59.010431 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:59.010821 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:59.011001 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:59.011769 (+ 768us) server_negotiation.cc:299] Negotiation successful
0806 02:48:59.012042 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":292,"threads_started":1}
I20260806 02:48:59.013603 16352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:58.992691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.67:56503)
0806 02:48:58.993603 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:58.993667 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:58.995943 (+ 2276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:58.996801 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:58.996815 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:58.997524 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:58.998242 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:58.998262 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:58.999555 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:58.999559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:59.000957 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:59.000972 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:59.001844 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:59.001850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:59.002093 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:59.003021 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:59.003048 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:59.005480 (+ 2432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:59.009119 (+ 3639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:59.009136 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:59.009159 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:59.009662 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:59.010063 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:59.010069 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:59.010073 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:59.010285 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:59.010964 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:59.010973 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:59.011378 (+ 405us) client_negotiation.cc:769] Sending connection context
0806 02:48:59.011689 (+ 311us) client_negotiation.cc:240] Negotiation successful
0806 02:48:59.012284 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":363,"threads_started":1}
I20260806 02:48:59.015271 16349 heartbeater.cc:347] Connected to a master server at 127.1.210.126:45859
I20260806 02:48:59.015632 16349 heartbeater.cc:464] Registering TS with master...
I20260806 02:48:59.016479 16349 heartbeater.cc:511] Master 127.1.210.126:45859 requested a full tablet report, sending...
I20260806 02:48:59.018167 15887 ts_manager.cc:194] Registered new tserver with Master: f24ecb82f42249ee97edfab14084c27c (127.1.210.67:44225)
I20260806 02:48:59.019057 15887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.67@KRBTEST.COM'} at 127.1.210.67:56503
I20260806 02:48:59.022172 1865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:48:59.041447 16351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:59.026969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35672 (local address 127.1.210.126:45859)
0806 02:48:59.027213 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:59.027218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:59.027678 (+ 460us) server_negotiation.cc:407] Connection header received
0806 02:48:59.027819 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:59.027823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:59.027891 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:59.027986 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:59.029454 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:59.030220 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:59.031838 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:59.032299 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:59.032412 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:59.035308 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:59.035313 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:59.035316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:59.035355 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:59.038217 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:59.039160 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:59.039164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:59.039166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:59.039318 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:59.039735 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:59.039747 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:59.039749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:59.040190 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:59.040412 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:59.040842 (+ 430us) server_negotiation.cc:299] Negotiation successful
0806 02:48:59.041172 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260806 02:48:59.045831 15887 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:48:59.052836 15887 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:48:59.076738 16364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:59.066546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55852 (local address 127.1.210.65:35323)
0806 02:48:59.067784 (+ 1238us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:59.067794 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:59.067823 (+ 29us) server_negotiation.cc:407] Connection header received
0806 02:48:59.067938 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:59.067946 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:59.068272 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:59.068529 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:59.069359 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:59.071028 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:59.073799 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:59.075361 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:59.075536 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:59.075802 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:59.076161 (+ 359us) server_negotiation.cc:299] Negotiation successful
0806 02:48:59.076341 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1049,"thread_start_us":1120,"threads_started":1}
I20260806 02:48:59.076828 16362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:59.066123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:35323 (local address 127.0.0.1:55852)
0806 02:48:59.067183 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:59.067234 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:59.067448 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:59.068541 (+ 1093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:59.068552 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:59.068602 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:59.069080 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:59.069098 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:59.071183 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:59.071190 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:59.073602 (+ 2412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:59.073619 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:59.075607 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:59.075614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:59.075898 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:59.075937 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 02:48:59.076112 (+ 175us) client_negotiation.cc:240] Negotiation successful
0806 02:48:59.076342 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"mutex_wait_us":27,"thread_start_us":464,"threads_started":1}
I20260806 02:48:59.079546 16363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:59.066220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:44225 (local address 127.0.0.1:50802)
0806 02:48:59.069548 (+ 3328us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:59.069564 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:59.069710 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:59.071640 (+ 1930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:59.071644 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:59.071673 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:59.072096 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:59.072121 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:59.074277 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:59.074283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:59.076257 (+ 1974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:59.076272 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:59.078599 (+ 2327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:59.078606 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:59.078821 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:59.078854 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 02:48:59.078956 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 02:48:59.079130 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3227,"mutex_wait_us":1,"thread_start_us":3091,"threads_started":1}
I20260806 02:48:59.079577 16365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:59.067232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50802 (local address 127.1.210.67:44225)
0806 02:48:59.069300 (+ 2068us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:59.069311 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:59.069664 (+ 353us) server_negotiation.cc:407] Connection header received
0806 02:48:59.071052 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:59.071066 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:59.071404 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:59.072227 (+ 823us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:59.072559 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:59.074103 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:59.076496 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:59.078398 (+ 1902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:59.078539 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:59.078822 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:59.079011 (+ 189us) server_negotiation.cc:299] Negotiation successful
0806 02:48:59.079194 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1895,"thread_start_us":1742,"threads_started":1}
I20260806 02:48:59.081301 16366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:59.069969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:45613 (local address 127.0.0.1:54136)
0806 02:48:59.070626 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:59.070650 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:59.070824 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:59.072373 (+ 1549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:59.072378 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:59.072406 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:59.072905 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:59.072918 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:59.075434 (+ 2516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:59.075441 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:59.078040 (+ 2599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:59.078057 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:59.080434 (+ 2377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:59.080441 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:59.080610 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:59.080654 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 02:48:59.080761 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 02:48:59.080912 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":303,"threads_started":1}
I20260806 02:48:59.081281 16014 tablet_service.cc:1467] Processing CreateTablet for tablet df90f51023e943069ffa3afd1fc09252 (DEFAULT_TABLE table=test-table [id=1f30250cea0b4f0896666ab98de40d90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:48:59.081974 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:59.070001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54136 (local address 127.1.210.66:45613)
0806 02:48:59.071725 (+ 1724us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:59.071735 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:59.071761 (+ 26us) server_negotiation.cc:407] Connection header received
0806 02:48:59.071863 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:59.071870 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:59.072147 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:59.072347 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:59.073099 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:59.075286 (+ 2187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:59.078268 (+ 2982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:59.079901 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:59.080074 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:59.080291 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:59.080837 (+ 546us) server_negotiation.cc:299] Negotiation successful
0806 02:48:59.081014 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1561,"thread_start_us":1465,"threads_started":1}
I20260806 02:48:59.083024 16014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df90f51023e943069ffa3afd1fc09252. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:48:59.084056 16284 tablet_service.cc:1467] Processing CreateTablet for tablet df90f51023e943069ffa3afd1fc09252 (DEFAULT_TABLE table=test-table [id=1f30250cea0b4f0896666ab98de40d90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:48:59.085608 16284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df90f51023e943069ffa3afd1fc09252. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:48:59.086079 16148 tablet_service.cc:1467] Processing CreateTablet for tablet df90f51023e943069ffa3afd1fc09252 (DEFAULT_TABLE table=test-table [id=1f30250cea0b4f0896666ab98de40d90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:48:59.088016 16148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df90f51023e943069ffa3afd1fc09252. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:48:59.099572 16368 tablet_bootstrap.cc:492] T df90f51023e943069ffa3afd1fc09252 P a2a25e0aaf014da8ab4bac97925b34d5: Bootstrap starting.
I20260806 02:48:59.103499 16369 tablet_bootstrap.cc:492] T df90f51023e943069ffa3afd1fc09252 P f24ecb82f42249ee97edfab14084c27c: Bootstrap starting.
I20260806 02:48:59.105077 16370 tablet_bootstrap.cc:492] T df90f51023e943069ffa3afd1fc09252 P 88480c29dea44647ba7cc9e90c927baa: Bootstrap starting.
I20260806 02:48:59.105686 16368 tablet_bootstrap.cc:654] T df90f51023e943069ffa3afd1fc09252 P a2a25e0aaf014da8ab4bac97925b34d5: Neither blocks nor log segments found. Creating new log.
I20260806 02:48:59.107167 16368 log.cc:824] T df90f51023e943069ffa3afd1fc09252 P a2a25e0aaf014da8ab4bac97925b34d5: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:59.107331 16369 tablet_bootstrap.cc:654] T df90f51023e943069ffa3afd1fc09252 P f24ecb82f42249ee97edfab14084c27c: Neither blocks nor log segments found. Creating new log.
I20260806 02:48:59.108532 16369 log.cc:824] T df90f51023e943069ffa3afd1fc09252 P f24ecb82f42249ee97edfab14084c27c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:59.108808 16370 tablet_bootstrap.cc:654] T df90f51023e943069ffa3afd1fc09252 P 88480c29dea44647ba7cc9e90c927baa: Neither blocks nor log segments found. Creating new log.
I20260806 02:48:59.109997 16370 log.cc:824] T df90f51023e943069ffa3afd1fc09252 P 88480c29dea44647ba7cc9e90c927baa: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:59.111148 16368 tablet_bootstrap.cc:492] T df90f51023e943069ffa3afd1fc09252 P a2a25e0aaf014da8ab4bac97925b34d5: No bootstrap required, opened a new log
I20260806 02:48:59.111457 16368 ts_tablet_manager.cc:1397] T df90f51023e943069ffa3afd1fc09252 P a2a25e0aaf014da8ab4bac97925b34d5: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260806 02:48:59.111650 16369 tablet_bootstrap.cc:492] T df90f51023e943069ffa3afd1fc09252 P f24ecb82f42249ee97edfab14084c27c: No bootstrap required, opened a new log
I20260806 02:48:59.111939 16369 ts_tablet_manager.cc:1397] T df90f51023e943069ffa3afd1fc09252 P f24ecb82f42249ee97edfab14084c27c: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260806 02:48:59.113761 16370 tablet_bootstrap.cc:492] T df90f51023e943069ffa3afd1fc09252 P 88480c29dea44647ba7cc9e90c927baa: No bootstrap required, opened a new log
I20260806 02:48:59.114048 16370 ts_tablet_manager.cc:1397] T df90f51023e943069ffa3afd1fc09252 P 88480c29dea44647ba7cc9e90c927baa: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260806 02:48:59.119374 16369 raft_consensus.cc:348] T df90f51023e943069ffa3afd1fc09252 P f24ecb82f42249ee97edfab14084c27c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a25e0aaf014da8ab4bac97925b34d5" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 35323 } } peers { permanent_uuid: "f24ecb82f42249ee97edfab14084c27c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44225 } } peers { permanent_uuid: "88480c29dea44647ba7cc9e90c927baa" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45613 } }
I20260806 02:48:59.119807 16369 raft_consensus.cc:374] T df90f51023e943069ffa3afd1fc09252 P f24ecb82f42249ee97edfab14084c27c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:48:59.119905 16369 raft_consensus.cc:729] T df90f51023e943069ffa3afd1fc09252 P f24ecb82f42249ee97edfab14084c27c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24ecb82f42249ee97edfab14084c27c, State: Initialized, Role: FOLLOWER
I20260806 02:48:59.120592 16369 consensus_queue.cc:260] T df90f51023e943069ffa3afd1fc09252 P f24ecb82f42249ee97edfab14084c27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a25e0aaf014da8ab4bac97925b34d5" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 35323 } } peers { permanent_uuid: "f24ecb82f42249ee97edfab14084c27c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44225 } } peers { permanent_uuid: "88480c29dea44647ba7cc9e90c927baa" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45613 } }
I20260806 02:48:59.122808 16368 raft_consensus.cc:348] T df90f51023e943069ffa3afd1fc09252 P a2a25e0aaf014da8ab4bac97925b34d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a25e0aaf014da8ab4bac97925b34d5" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 35323 } } peers { permanent_uuid: "f24ecb82f42249ee97edfab14084c27c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44225 } } peers { permanent_uuid: "88480c29dea44647ba7cc9e90c927baa" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45613 } }
I20260806 02:48:59.123386 16368 raft_consensus.cc:374] T df90f51023e943069ffa3afd1fc09252 P a2a25e0aaf014da8ab4bac97925b34d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:48:59.123484 16368 raft_consensus.cc:729] T df90f51023e943069ffa3afd1fc09252 P a2a25e0aaf014da8ab4bac97925b34d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a25e0aaf014da8ab4bac97925b34d5, State: Initialized, Role: FOLLOWER
I20260806 02:48:59.123946 16349 heartbeater.cc:503] Master 127.1.210.126:45859 was elected leader, sending a full tablet report...
I20260806 02:48:59.124214 16368 consensus_queue.cc:260] T df90f51023e943069ffa3afd1fc09252 P a2a25e0aaf014da8ab4bac97925b34d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a25e0aaf014da8ab4bac97925b34d5" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 35323 } } peers { permanent_uuid: "f24ecb82f42249ee97edfab14084c27c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44225 } } peers { permanent_uuid: "88480c29dea44647ba7cc9e90c927baa" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45613 } }
I20260806 02:48:59.124529 16369 ts_tablet_manager.cc:1428] T df90f51023e943069ffa3afd1fc09252 P f24ecb82f42249ee97edfab14084c27c: Time spent starting tablet: real 0.012s user 0.002s sys 0.011s
I20260806 02:48:59.125996 16370 raft_consensus.cc:348] T df90f51023e943069ffa3afd1fc09252 P 88480c29dea44647ba7cc9e90c927baa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a25e0aaf014da8ab4bac97925b34d5" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 35323 } } peers { permanent_uuid: "f24ecb82f42249ee97edfab14084c27c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44225 } } peers { permanent_uuid: "88480c29dea44647ba7cc9e90c927baa" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45613 } }
I20260806 02:48:59.126489 16370 raft_consensus.cc:374] T df90f51023e943069ffa3afd1fc09252 P 88480c29dea44647ba7cc9e90c927baa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:48:59.126595 16370 raft_consensus.cc:729] T df90f51023e943069ffa3afd1fc09252 P 88480c29dea44647ba7cc9e90c927baa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88480c29dea44647ba7cc9e90c927baa, State: Initialized, Role: FOLLOWER
I20260806 02:48:59.127278 16370 consensus_queue.cc:260] T df90f51023e943069ffa3afd1fc09252 P 88480c29dea44647ba7cc9e90c927baa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a25e0aaf014da8ab4bac97925b34d5" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 35323 } } peers { permanent_uuid: "f24ecb82f42249ee97edfab14084c27c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44225 } } peers { permanent_uuid: "88480c29dea44647ba7cc9e90c927baa" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45613 } }
I20260806 02:48:59.130012 16215 heartbeater.cc:503] Master 127.1.210.126:45859 was elected leader, sending a full tablet report...
I20260806 02:48:59.130956 16370 ts_tablet_manager.cc:1428] T df90f51023e943069ffa3afd1fc09252 P 88480c29dea44647ba7cc9e90c927baa: Time spent starting tablet: real 0.017s user 0.014s sys 0.004s
I20260806 02:48:59.132783 16368 ts_tablet_manager.cc:1428] T df90f51023e943069ffa3afd1fc09252 P a2a25e0aaf014da8ab4bac97925b34d5: Time spent starting tablet: real 0.021s user 0.016s sys 0.003s
I20260806 02:48:59.143541 16375 raft_consensus.cc:482] T df90f51023e943069ffa3afd1fc09252 P a2a25e0aaf014da8ab4bac97925b34d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:48:59.143867 16375 raft_consensus.cc:504] T df90f51023e943069ffa3afd1fc09252 P a2a25e0aaf014da8ab4bac97925b34d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a25e0aaf014da8ab4bac97925b34d5" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 35323 } } peers { permanent_uuid: "f24ecb82f42249ee97edfab14084c27c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44225 } } peers { permanent_uuid: "88480c29dea44647ba7cc9e90c927baa" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45613 } }
I20260806 02:48:59.145809 16375 leader_election.cc:288] T df90f51023e943069ffa3afd1fc09252 P a2a25e0aaf014da8ab4bac97925b34d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f24ecb82f42249ee97edfab14084c27c (127.1.210.67:44225), 88480c29dea44647ba7cc9e90c927baa (127.1.210.66:45613)
I20260806 02:48:59.153542 16365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:59.146190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:54117 (local address 127.1.210.67:44225)
0806 02:48:59.146391 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:59.146399 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:59.146805 (+ 406us) server_negotiation.cc:407] Connection header received
0806 02:48:59.147113 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:59.147119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:59.147203 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:59.147314 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:59.148069 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:59.149049 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:59.151422 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:59.152601 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:59.152777 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:59.152865 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:59.153088 (+ 223us) server_negotiation.cc:299] Negotiation successful
0806 02:48:59.153238 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 02:48:59.153555 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:59.145996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:44225 (local address 127.1.210.65:54117)
0806 02:48:59.146700 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:59.146722 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:59.146924 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:59.147462 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:59.147478 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:59.147506 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:59.147931 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:59.147941 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:59.149174 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:59.149181 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:59.151213 (+ 2032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:59.151228 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:59.152800 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:59.152804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:59.152924 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:59.152949 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 02:48:59.153031 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 02:48:59.153137 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":370,"threads_started":1}
I20260806 02:48:59.154119 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:59.147062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:45545 (local address 127.1.210.66:45613)
0806 02:48:59.147267 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:59.147275 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:59.147690 (+ 415us) server_negotiation.cc:407] Connection header received
0806 02:48:59.147860 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:59.147865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:59.147937 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:59.148023 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:59.148653 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:59.149727 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:59.151903 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:59.153137 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:59.153281 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:59.153354 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:59.153774 (+ 420us) server_negotiation.cc:299] Negotiation successful
0806 02:48:59.153933 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 02:48:59.154860 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:59.146863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:45613 (local address 127.1.210.65:45545)
0806 02:48:59.147575 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:59.147593 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:59.147726 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:59.148099 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:59.148103 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:59.148132 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:59.148490 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:59.148501 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:59.149885 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:59.149890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:59.151745 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:59.151755 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:59.153361 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:59.153368 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:59.153558 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:59.153601 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 02:48:59.154314 (+ 713us) client_negotiation.cc:240] Negotiation successful
0806 02:48:59.154488 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":367,"threads_started":1}
I20260806 02:48:59.155017 16304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df90f51023e943069ffa3afd1fc09252" candidate_uuid: "a2a25e0aaf014da8ab4bac97925b34d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24ecb82f42249ee97edfab14084c27c" is_pre_election: true
I20260806 02:48:59.155673 16304 raft_consensus.cc:2454] T df90f51023e943069ffa3afd1fc09252 P f24ecb82f42249ee97edfab14084c27c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a25e0aaf014da8ab4bac97925b34d5 in term 0.
I20260806 02:48:59.156015 16170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df90f51023e943069ffa3afd1fc09252" candidate_uuid: "a2a25e0aaf014da8ab4bac97925b34d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88480c29dea44647ba7cc9e90c927baa" is_pre_election: true
I20260806 02:48:59.156443 15966 leader_election.cc:302] T df90f51023e943069ffa3afd1fc09252 P a2a25e0aaf014da8ab4bac97925b34d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a25e0aaf014da8ab4bac97925b34d5, f24ecb82f42249ee97edfab14084c27c; no voters:
I20260806 02:48:59.156554 16170 raft_consensus.cc:2454] T df90f51023e943069ffa3afd1fc09252 P 88480c29dea44647ba7cc9e90c927baa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a25e0aaf014da8ab4bac97925b34d5 in term 0.
I20260806 02:48:59.156922 16375 raft_consensus.cc:2781] T df90f51023e943069ffa3afd1fc09252 P a2a25e0aaf014da8ab4bac97925b34d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:48:59.157111 16375 raft_consensus.cc:482] T df90f51023e943069ffa3afd1fc09252 P a2a25e0aaf014da8ab4bac97925b34d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:48:59.157229 16375 raft_consensus.cc:3037] T df90f51023e943069ffa3afd1fc09252 P a2a25e0aaf014da8ab4bac97925b34d5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:48:59.159616 16375 raft_consensus.cc:504] T df90f51023e943069ffa3afd1fc09252 P a2a25e0aaf014da8ab4bac97925b34d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a25e0aaf014da8ab4bac97925b34d5" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 35323 } } peers { permanent_uuid: "f24ecb82f42249ee97edfab14084c27c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44225 } } peers { permanent_uuid: "88480c29dea44647ba7cc9e90c927baa" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45613 } }
I20260806 02:48:59.160283 16375 leader_election.cc:288] T df90f51023e943069ffa3afd1fc09252 P a2a25e0aaf014da8ab4bac97925b34d5 [CANDIDATE]: Term 1 election: Requested vote from peers f24ecb82f42249ee97edfab14084c27c (127.1.210.67:44225), 88480c29dea44647ba7cc9e90c927baa (127.1.210.66:45613)
I20260806 02:48:59.160779 16304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df90f51023e943069ffa3afd1fc09252" candidate_uuid: "a2a25e0aaf014da8ab4bac97925b34d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f24ecb82f42249ee97edfab14084c27c"
I20260806 02:48:59.160825 16170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df90f51023e943069ffa3afd1fc09252" candidate_uuid: "a2a25e0aaf014da8ab4bac97925b34d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88480c29dea44647ba7cc9e90c927baa"
I20260806 02:48:59.160960 16304 raft_consensus.cc:3037] T df90f51023e943069ffa3afd1fc09252 P f24ecb82f42249ee97edfab14084c27c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:48:59.161156 16170 raft_consensus.cc:3037] T df90f51023e943069ffa3afd1fc09252 P 88480c29dea44647ba7cc9e90c927baa [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:48:59.163378 16304 raft_consensus.cc:2454] T df90f51023e943069ffa3afd1fc09252 P f24ecb82f42249ee97edfab14084c27c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a25e0aaf014da8ab4bac97925b34d5 in term 1.
I20260806 02:48:59.163635 16170 raft_consensus.cc:2454] T df90f51023e943069ffa3afd1fc09252 P 88480c29dea44647ba7cc9e90c927baa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a25e0aaf014da8ab4bac97925b34d5 in term 1.
I20260806 02:48:59.164022 15966 leader_election.cc:302] T df90f51023e943069ffa3afd1fc09252 P a2a25e0aaf014da8ab4bac97925b34d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a25e0aaf014da8ab4bac97925b34d5, f24ecb82f42249ee97edfab14084c27c; no voters:
I20260806 02:48:59.164351 16375 raft_consensus.cc:2781] T df90f51023e943069ffa3afd1fc09252 P a2a25e0aaf014da8ab4bac97925b34d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:48:59.164820 16375 raft_consensus.cc:686] T df90f51023e943069ffa3afd1fc09252 P a2a25e0aaf014da8ab4bac97925b34d5 [term 1 LEADER]: Becoming Leader. State: Replica: a2a25e0aaf014da8ab4bac97925b34d5, State: Running, Role: LEADER
I20260806 02:48:59.165306 16375 consensus_queue.cc:237] T df90f51023e943069ffa3afd1fc09252 P a2a25e0aaf014da8ab4bac97925b34d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a25e0aaf014da8ab4bac97925b34d5" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 35323 } } peers { permanent_uuid: "f24ecb82f42249ee97edfab14084c27c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44225 } } peers { permanent_uuid: "88480c29dea44647ba7cc9e90c927baa" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45613 } }
I20260806 02:48:59.170369 15886 catalog_manager.cc:5495] T df90f51023e943069ffa3afd1fc09252 P a2a25e0aaf014da8ab4bac97925b34d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a25e0aaf014da8ab4bac97925b34d5 (127.1.210.65). New cstate: current_term: 1 leader_uuid: "a2a25e0aaf014da8ab4bac97925b34d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a25e0aaf014da8ab4bac97925b34d5" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 35323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f24ecb82f42249ee97edfab14084c27c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88480c29dea44647ba7cc9e90c927baa" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45613 } health_report { overall_health: UNKNOWN } } }
W20260806 02:48:59.189626 16216 tablet.cc:2369] T df90f51023e943069ffa3afd1fc09252 P 88480c29dea44647ba7cc9e90c927baa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:48:59.190306 16364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:59.184665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55866 (local address 127.1.210.65:35323)
0806 02:48:59.184863 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:59.184868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:59.184887 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:48:59.185019 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:59.185023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:59.185123 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:59.185256 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:48:59.185677 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:59.186454 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:59.188035 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:59.188517 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:59.188623 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:59.189138 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:48:59.189691 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:48:59.189873 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:59.190020 (+ 147us) server_negotiation.cc:299] Negotiation successful
0806 02:48:59.190096 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 02:48:59.199375 16170 raft_consensus.cc:1261] T df90f51023e943069ffa3afd1fc09252 P 88480c29dea44647ba7cc9e90c927baa [term 1 FOLLOWER]: Refusing update from remote peer a2a25e0aaf014da8ab4bac97925b34d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:48:59.199363 16304 raft_consensus.cc:1261] T df90f51023e943069ffa3afd1fc09252 P f24ecb82f42249ee97edfab14084c27c [term 1 FOLLOWER]: Refusing update from remote peer a2a25e0aaf014da8ab4bac97925b34d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:48:59.200417 16380 consensus_queue.cc:1035] T df90f51023e943069ffa3afd1fc09252 P a2a25e0aaf014da8ab4bac97925b34d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88480c29dea44647ba7cc9e90c927baa" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:48:59.200877 16375 consensus_queue.cc:1035] T df90f51023e943069ffa3afd1fc09252 P a2a25e0aaf014da8ab4bac97925b34d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f24ecb82f42249ee97edfab14084c27c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:48:59.216178 16384 mvcc.cc:205] Tried to move back new op lower bound from 7315392672548945920 to 7315392672424800256. Current Snapshot: MvccSnapshot[applied={T|T < 7315392672548945920}]
I20260806 02:48:59.237604 15887 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:48:59.237998 15887 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:48:59.242234 15887 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80: Sending DeleteTablet for 3 replicas of tablet df90f51023e943069ffa3afd1fc09252
I20260806 02:48:59.243708 16014 tablet_service.cc:1514] Processing DeleteTablet for tablet df90f51023e943069ffa3afd1fc09252 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:48:59 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:55852
I20260806 02:48:59.243701 16284 tablet_service.cc:1514] Processing DeleteTablet for tablet df90f51023e943069ffa3afd1fc09252 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:48:59 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:50802
I20260806 02:48:59.243913 16148 tablet_service.cc:1514] Processing DeleteTablet for tablet df90f51023e943069ffa3afd1fc09252 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:48:59 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:54136
I20260806 02:48:59.244496 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 15949
I20260806 02:48:59.256085 16394 tablet_replica.cc:330] T df90f51023e943069ffa3afd1fc09252 P a2a25e0aaf014da8ab4bac97925b34d5: stopping tablet replica
I20260806 02:48:59.256681 16394 raft_consensus.cc:2229] T df90f51023e943069ffa3afd1fc09252 P a2a25e0aaf014da8ab4bac97925b34d5 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:48:59.257535 16394 raft_consensus.cc:2258] T df90f51023e943069ffa3afd1fc09252 P a2a25e0aaf014da8ab4bac97925b34d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:48:59.258396 16395 tablet_replica.cc:330] T df90f51023e943069ffa3afd1fc09252 P 88480c29dea44647ba7cc9e90c927baa: stopping tablet replica
I20260806 02:48:59.260367 16394 ts_tablet_manager.cc:1905] T df90f51023e943069ffa3afd1fc09252 P a2a25e0aaf014da8ab4bac97925b34d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:48:59.260795 16395 raft_consensus.cc:2229] T df90f51023e943069ffa3afd1fc09252 P 88480c29dea44647ba7cc9e90c927baa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:48:59.261431 16393 tablet_replica.cc:330] T df90f51023e943069ffa3afd1fc09252 P f24ecb82f42249ee97edfab14084c27c: stopping tablet replica
I20260806 02:48:59.261466 16395 raft_consensus.cc:2258] T df90f51023e943069ffa3afd1fc09252 P 88480c29dea44647ba7cc9e90c927baa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:48:59.262111 16393 raft_consensus.cc:2229] T df90f51023e943069ffa3afd1fc09252 P f24ecb82f42249ee97edfab14084c27c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:48:59.262666 16393 raft_consensus.cc:2258] T df90f51023e943069ffa3afd1fc09252 P f24ecb82f42249ee97edfab14084c27c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:48:59.265105 16393 ts_tablet_manager.cc:1905] T df90f51023e943069ffa3afd1fc09252 P f24ecb82f42249ee97edfab14084c27c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:48:59.268810 16395 ts_tablet_manager.cc:1905] T df90f51023e943069ffa3afd1fc09252 P 88480c29dea44647ba7cc9e90c927baa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:48:59.270879 16394 ts_tablet_manager.cc:1918] T df90f51023e943069ffa3afd1fc09252 P a2a25e0aaf014da8ab4bac97925b34d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:48:59.271118 16394 log.cc:1196] T df90f51023e943069ffa3afd1fc09252 P a2a25e0aaf014da8ab4bac97925b34d5: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/wal/wals/df90f51023e943069ffa3afd1fc09252
I20260806 02:48:59.271835 16394 ts_tablet_manager.cc:1939] T df90f51023e943069ffa3afd1fc09252 P a2a25e0aaf014da8ab4bac97925b34d5: Deleting consensus metadata
I20260806 02:48:59.272392 16393 ts_tablet_manager.cc:1918] T df90f51023e943069ffa3afd1fc09252 P f24ecb82f42249ee97edfab14084c27c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:48:59.272588 16393 log.cc:1196] T df90f51023e943069ffa3afd1fc09252 P f24ecb82f42249ee97edfab14084c27c: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/wal/wals/df90f51023e943069ffa3afd1fc09252
I20260806 02:48:59.273175 16393 ts_tablet_manager.cc:1939] T df90f51023e943069ffa3afd1fc09252 P f24ecb82f42249ee97edfab14084c27c: Deleting consensus metadata
I20260806 02:48:59.274042 15873 catalog_manager.cc:4841] TS a2a25e0aaf014da8ab4bac97925b34d5 (127.1.210.65:35323): tablet df90f51023e943069ffa3afd1fc09252 (table test-table [id=1f30250cea0b4f0896666ab98de40d90]) successfully deleted
I20260806 02:48:59.275072 15872 catalog_manager.cc:4841] TS f24ecb82f42249ee97edfab14084c27c (127.1.210.67:44225): tablet df90f51023e943069ffa3afd1fc09252 (table test-table [id=1f30250cea0b4f0896666ab98de40d90]) successfully deleted
Aug 06 02:48:59 dist-test-slave-8tdl krb5kdc[15840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.65@KRBTEST.COM
I20260806 02:48:59.280218 16395 ts_tablet_manager.cc:1918] T df90f51023e943069ffa3afd1fc09252 P 88480c29dea44647ba7cc9e90c927baa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:48:59.280424 16395 log.cc:1196] T df90f51023e943069ffa3afd1fc09252 P 88480c29dea44647ba7cc9e90c927baa: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/wal/wals/df90f51023e943069ffa3afd1fc09252
I20260806 02:48:59.281069 16395 ts_tablet_manager.cc:1939] T df90f51023e943069ffa3afd1fc09252 P 88480c29dea44647ba7cc9e90c927baa: Deleting consensus metadata
I20260806 02:48:59.283106 15872 catalog_manager.cc:4841] TS 88480c29dea44647ba7cc9e90c927baa (127.1.210.66:45613): tablet df90f51023e943069ffa3afd1fc09252 (table test-table [id=1f30250cea0b4f0896666ab98de40d90]) successfully deleted
I20260806 02:48:59.288158 16364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:59.246093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55874 (local address 127.1.210.65:35323)
0806 02:48:59.246366 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:59.246373 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:59.255315 (+ 8942us) server_negotiation.cc:407] Connection header received
0806 02:48:59.256094 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:59.256100 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:59.256221 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:59.256313 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:59.261939 (+ 5626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:59.263432 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:59.269112 (+ 5680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:59.269887 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:59.270012 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:59.280092 (+ 10080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:59.280115 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:59.280137 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:59.280193 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:59.284542 (+ 4349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:59.285474 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:59.285485 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:59.285488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:59.285697 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:59.286184 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:59.286188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:59.286190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:59.286598 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:59.287003 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:59.287440 (+ 437us) server_negotiation.cc:299] Negotiation successful
0806 02:48:59.287709 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 02:48:59.289222 16074 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:48:59.395560 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 15949
I20260806 02:48:59.418211 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 16085
Aug 06 02:48:59 dist-test-slave-8tdl krb5kdc[15840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.66@KRBTEST.COM
I20260806 02:48:59.440135 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:59.419094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54140 (local address 127.1.210.66:45613)
0806 02:48:59.419282 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:59.419287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:59.419306 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:48:59.419462 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:59.419466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:59.419544 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:59.419631 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:59.421039 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:59.421809 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:59.423234 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:59.423711 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:59.423828 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:59.433665 (+ 9837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:59.433683 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:59.433700 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:59.433742 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:59.436958 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:59.437915 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:59.437922 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:59.437924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:59.438048 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:59.438495 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:59.438498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:59.438500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:59.439000 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:59.439226 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:59.439643 (+ 417us) server_negotiation.cc:299] Negotiation successful
0806 02:48:59.439846 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 02:48:59.441067 16210 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:48:59.534353 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 16085
I20260806 02:48:59.556689 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 16221
Aug 06 02:48:59 dist-test-slave-8tdl krb5kdc[15840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.67@KRBTEST.COM
I20260806 02:48:59.573298 16365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:59.557694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50812 (local address 127.1.210.67:44225)
0806 02:48:59.557917 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:59.557924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:59.558005 (+ 81us) server_negotiation.cc:407] Connection header received
0806 02:48:59.558264 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:59.558270 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:59.558358 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:59.558481 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:59.559904 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:59.560688 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:59.561938 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:59.562373 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:59.562511 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:59.567468 (+ 4957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:59.567485 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:59.567503 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:59.567550 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:59.570350 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:59.571265 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:59.571272 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:59.571275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:59.571449 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:59.571897 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:59.571901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:59.571904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:59.572287 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:59.572525 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:59.572869 (+ 344us) server_negotiation.cc:299] Negotiation successful
0806 02:48:59.573043 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 02:48:59.574045 16344 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:48:59.665911 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 16221
I20260806 02:48:59.683848 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 15854
I20260806 02:48:59.685073 15917 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:48:59.803670 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 15854
I20260806 02:48:59.830349 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:45859
--webserver_interface=127.1.210.126
--webserver_port=35143
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:45859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:49:00.192250 16401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:00.192587 16401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:00.192777 16401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:00.200520 16401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:49:00.200649 16401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:00.200698 16401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:00.200816 16401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:49:00.200906 16401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:49:00.211630 16401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:45859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:45859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=35143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:49:00.213166 16401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:00.215188 16401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:00.244781 16408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:00.244848 16407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:00.244801 16410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:00.248869 16401 server_base.cc:1034] running on GCE node
I20260806 02:49:00.249924 16401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:00.251972 16401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:00.253213 16401 hybrid_clock.cc:648] HybridClock initialized: now 1785984540253158 us; error 41 us; skew 500 ppm
I20260806 02:49:00.253664 16401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:49:00 dist-test-slave-8tdl krb5kdc[15840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984540, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:00.261518 16401 init.cc:377] Logged in from keytab as kudu/127.1.210.126@KRBTEST.COM (short username kudu)
I20260806 02:49:00.263586 16401 webserver.cc:460] Webserver started at http://127.1.210.126:35143/ using document root <none> and password file <none>
I20260806 02:49:00.264498 16401 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:00.264688 16401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:00.271595 16401 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:49:00.274992 16417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:00.276599 16401 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:49:00.276800 16401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "1b645c239c754d49a17f96a216c4af80"
format_stamp: "Formatted at 2026-08-06 02:48:57 on dist-test-slave-8tdl"
server_key: "80246ef024936cd5ac4eb7aed35dc814"
server_key_iv: "267a2538434db8f71d260a2b8d04f70e"
server_key_version: "encryptionkey@0"
I20260806 02:49:00.277997 16401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:00.328857 16401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:00.334039 16401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:00.334383 16401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:00.356242 16401 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:45859
I20260806 02:49:00.356287 16469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:45859 every 8 connection(s)
I20260806 02:49:00.358017 16401 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:49:00.360994 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 16401
I20260806 02:49:00.362092 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.65:35323
--local_ip_for_outbound_sockets=127.1.210.65
--tserver_master_addrs=127.1.210.126:45859
--webserver_port=44417
--webserver_interface=127.1.210.65
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:49:00.368189 16470 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:49:00.373556 16470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80: Bootstrap starting.
I20260806 02:49:00.395560 16470 log.cc:824] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:00.407981 16470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:49:00.408617 16470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80: Bootstrap complete.
I20260806 02:49:00.418319 16470 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b645c239c754d49a17f96a216c4af80" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45859 } }
I20260806 02:49:00.419353 16470 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b645c239c754d49a17f96a216c4af80, State: Initialized, Role: FOLLOWER
I20260806 02:49:00.420095 16470 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b645c239c754d49a17f96a216c4af80" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45859 } }
I20260806 02:49:00.420361 16470 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:49:00.420500 16470 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:49:00.420689 16470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:49:00.424846 16470 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b645c239c754d49a17f96a216c4af80" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45859 } }
I20260806 02:49:00.425427 16470 leader_election.cc:302] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b645c239c754d49a17f96a216c4af80; no voters:
I20260806 02:49:00.426124 16470 leader_election.cc:288] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 02:49:00.426271 16474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 02:49:00.427551 16474 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [term 2 LEADER]: Becoming Leader. State: Replica: 1b645c239c754d49a17f96a216c4af80, State: Running, Role: LEADER
I20260806 02:49:00.428228 16474 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b645c239c754d49a17f96a216c4af80" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45859 } }
I20260806 02:49:00.428673 16470 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:49:00.430920 16476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b645c239c754d49a17f96a216c4af80. Latest consensus state: current_term: 2 leader_uuid: "1b645c239c754d49a17f96a216c4af80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b645c239c754d49a17f96a216c4af80" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45859 } } }
I20260806 02:49:00.431347 16476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:00.431938 16475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1b645c239c754d49a17f96a216c4af80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b645c239c754d49a17f96a216c4af80" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45859 } } }
I20260806 02:49:00.432268 16475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:00.443082 16479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:49:00.451524 16479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:49:00.453397 16479 catalog_manager.cc:1259] Loaded cluster ID: 67ed54e8ea694172866975565c9c0c1b
I20260806 02:49:00.453600 16479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:49:00.457379 16479 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260806 02:49:00.475724 16479 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260806 02:49:00.489959 16479 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:49:00.492535 16479 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80: Loaded TSK: 0
I20260806 02:49:00.494441 16479 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 02:49:00.753415 16472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:00.753772 16472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:00.753932 16472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:00.761591 16472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:00.761761 16472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:00.761938 16472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.65
I20260806 02:49:00.772790 16472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.65:35323
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.65
--webserver_port=44417
--tserver_master_addrs=127.1.210.126:45859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.65
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:49:00.774396 16472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:00.775892 16472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:00.792622 16499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:00.792603 16501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:00.793463 16498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:00.794296 16472 server_base.cc:1034] running on GCE node
I20260806 02:49:00.795265 16472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:00.796747 16472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:00.798045 16472 hybrid_clock.cc:648] HybridClock initialized: now 1785984540797953 us; error 67 us; skew 500 ppm
I20260806 02:49:00.798506 16472 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:49:00 dist-test-slave-8tdl krb5kdc[15840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984540, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:00.803159 16472 init.cc:377] Logged in from keytab as kudu/127.1.210.65@KRBTEST.COM (short username kudu)
I20260806 02:49:00.804854 16472 webserver.cc:460] Webserver started at http://127.1.210.65:44417/ using document root <none> and password file <none>
I20260806 02:49:00.805639 16472 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:00.805787 16472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:00.811967 16472 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:49:00.815231 16508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:00.816882 16472 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 02:49:00.817124 16472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/wal
uuid: "a2a25e0aaf014da8ab4bac97925b34d5"
format_stamp: "Formatted at 2026-08-06 02:48:57 on dist-test-slave-8tdl"
server_key: "8e9a6c4da6256d24c0a019d170b7fbe8"
server_key_iv: "d9f8cfd24fdf0d01605fb2199f920d70"
server_key_version: "encryptionkey@0"
I20260806 02:49:00.818190 16472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:00.849323 16472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:00.854355 16472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:00.854655 16472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:00.855901 16472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:49:00.857358 16472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:49:00.857481 16472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:00.857615 16472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:49:00.857709 16472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:00.896596 16472 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.65:35323
I20260806 02:49:00.896687 16621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.65:35323 every 8 connection(s)
I20260806 02:49:00.898137 16472 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
I20260806 02:49:00.903936 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 16472
I20260806 02:49:00.904790 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.66:45613
--local_ip_for_outbound_sockets=127.1.210.66
--tserver_master_addrs=127.1.210.126:45859
--webserver_port=34855
--webserver_interface=127.1.210.66
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:49:00 dist-test-slave-8tdl krb5kdc[15840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984540, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:49:00.925554 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:00.901059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:41343 (local address 127.1.210.126:45859)
0806 02:49:00.901682 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:00.901694 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:00.902299 (+ 605us) server_negotiation.cc:407] Connection header received
0806 02:49:00.904795 (+ 2496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:00.904816 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:00.905329 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:00.905759 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:00.907421 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:00.909461 (+ 2040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:00.911164 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:00.912322 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:00.912549 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:00.916414 (+ 3865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:00.916423 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:00.916449 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:00.916496 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:00.919775 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:00.920917 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:00.920925 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:00.920931 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:00.921133 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:00.921767 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:00.921771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:00.921773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:00.922322 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:00.922529 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:00.923678 (+ 1149us) server_negotiation.cc:299] Negotiation successful
0806 02:49:00.924156 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":293,"threads_started":1}
I20260806 02:49:00.925586 16625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:00.901229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.65:41343)
0806 02:49:00.902069 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:00.902125 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:00.904050 (+ 1925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:00.905913 (+ 1863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:00.905921 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:00.906481 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:49:00.907165 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:00.907184 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:00.909663 (+ 2479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:00.909666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:00.911015 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:00.911028 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:00.912552 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:00.912557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:00.912772 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:00.913514 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:49:00.913537 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:49:00.916047 (+ 2510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:49:00.919968 (+ 3921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:00.919981 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:00.920003 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:00.920721 (+ 718us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:00.921270 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:00.921278 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:00.921285 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:00.921558 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:00.922492 (+ 934us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:49:00.922502 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:49:00.923009 (+ 507us) client_negotiation.cc:769] Sending connection context
0806 02:49:00.923382 (+ 373us) client_negotiation.cc:240] Negotiation successful
0806 02:49:00.924034 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":366,"threads_started":1}
I20260806 02:49:00.929395 16622 heartbeater.cc:347] Connected to a master server at 127.1.210.126:45859
I20260806 02:49:00.929862 16622 heartbeater.cc:464] Registering TS with master...
I20260806 02:49:00.930855 16622 heartbeater.cc:511] Master 127.1.210.126:45859 requested a full tablet report, sending...
I20260806 02:49:00.933672 16434 ts_manager.cc:194] Registered new tserver with Master: a2a25e0aaf014da8ab4bac97925b34d5 (127.1.210.65:35323)
I20260806 02:49:00.936153 16434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.65@KRBTEST.COM'} at 127.1.210.65:41343
W20260806 02:49:01.261807 16626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:01.262107 16626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:01.262203 16626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:01.269766 16626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:01.269887 16626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:01.270035 16626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.66
I20260806 02:49:01.280086 16626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.66:45613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.66
--webserver_port=34855
--tserver_master_addrs=127.1.210.126:45859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.66
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:49:01.281505 16626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:01.283041 16626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:01.297385 16631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:01.297488 16634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:01.297458 16632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:01.298127 16626 server_base.cc:1034] running on GCE node
I20260806 02:49:01.299150 16626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:01.300257 16626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:01.301518 16626 hybrid_clock.cc:648] HybridClock initialized: now 1785984541301458 us; error 54 us; skew 500 ppm
I20260806 02:49:01.301867 16626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:49:01 dist-test-slave-8tdl krb5kdc[15840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984541, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:01.305307 16626 init.cc:377] Logged in from keytab as kudu/127.1.210.66@KRBTEST.COM (short username kudu)
I20260806 02:49:01.307072 16626 webserver.cc:460] Webserver started at http://127.1.210.66:34855/ using document root <none> and password file <none>
I20260806 02:49:01.307823 16626 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:01.307992 16626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:01.313587 16626 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260806 02:49:01.316540 16642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:01.318105 16626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:49:01.318308 16626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/wal
uuid: "88480c29dea44647ba7cc9e90c927baa"
format_stamp: "Formatted at 2026-08-06 02:48:58 on dist-test-slave-8tdl"
server_key: "c296c893bba7c99aa5b7ed43e32966e4"
server_key_iv: "ba3458ecfafe0798a0e8692548627204"
server_key_version: "encryptionkey@0"
I20260806 02:49:01.319458 16626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:01.336290 16626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:01.341184 16626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:01.341512 16626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:01.342622 16626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:49:01.344092 16626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:49:01.344221 16626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:01.344359 16626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:49:01.344447 16626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:01.383183 16626 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.66:45613
I20260806 02:49:01.383247 16756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.66:45613 every 8 connection(s)
I20260806 02:49:01.384662 16626 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
I20260806 02:49:01.387400 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 16626
I20260806 02:49:01.388116 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.67:44225
--local_ip_for_outbound_sockets=127.1.210.67
--tserver_master_addrs=127.1.210.126:45859
--webserver_port=38321
--webserver_interface=127.1.210.67
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:49:01 dist-test-slave-8tdl krb5kdc[15840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984541, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:49:01.407243 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:01.387514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:34197 (local address 127.1.210.126:45859)
0806 02:49:01.390924 (+ 3410us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:01.390933 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:01.390963 (+ 30us) server_negotiation.cc:407] Connection header received
0806 02:49:01.391177 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:01.391185 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:01.391300 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:01.391434 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:01.393029 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:01.394224 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:01.396015 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:01.396765 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:01.396919 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:01.401078 (+ 4159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:01.401083 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:01.401087 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:01.401124 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:01.403842 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:01.404679 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:01.404684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:01.404686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:01.404820 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:01.405300 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:01.405304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:01.405306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:01.405642 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:01.405803 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:01.406578 (+ 775us) server_negotiation.cc:299] Negotiation successful
0806 02:49:01.406878 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3231}
I20260806 02:49:01.408334 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:01.388037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.66:34197)
0806 02:49:01.388970 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:01.389029 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:01.390890 (+ 1861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:01.391649 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:01.391658 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:01.392234 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:49:01.392819 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:01.392838 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:01.394423 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:01.394427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:01.395832 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:01.395845 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:01.397002 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:01.397007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:01.397225 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:01.398014 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:49:01.398039 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:49:01.400767 (+ 2728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:49:01.404016 (+ 3249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:01.404027 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:01.404045 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:01.404534 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:01.404947 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:01.404953 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:01.404957 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:01.405163 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:01.405776 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:49:01.405784 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:49:01.406179 (+ 395us) client_negotiation.cc:769] Sending connection context
0806 02:49:01.406471 (+ 292us) client_negotiation.cc:240] Negotiation successful
0806 02:49:01.407095 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":428,"threads_started":1}
I20260806 02:49:01.410128 16757 heartbeater.cc:347] Connected to a master server at 127.1.210.126:45859
I20260806 02:49:01.410511 16757 heartbeater.cc:464] Registering TS with master...
I20260806 02:49:01.411551 16757 heartbeater.cc:511] Master 127.1.210.126:45859 requested a full tablet report, sending...
I20260806 02:49:01.413419 16434 ts_manager.cc:194] Registered new tserver with Master: 88480c29dea44647ba7cc9e90c927baa (127.1.210.66:45613)
I20260806 02:49:01.414340 16434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.66@KRBTEST.COM'} at 127.1.210.66:34197
W20260806 02:49:01.741977 16761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:01.742318 16761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:01.742476 16761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:01.750422 16761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:01.750554 16761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:01.750761 16761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.67
I20260806 02:49:01.761185 16761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.67:44225
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.67
--webserver_port=38321
--tserver_master_addrs=127.1.210.126:45859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.67
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:49:01.762640 16761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:01.764143 16761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:01.778589 16769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:01.778618 16767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:01.778661 16766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:01.782876 16761 server_base.cc:1034] running on GCE node
I20260806 02:49:01.783720 16761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:01.784826 16761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:01.786080 16761 hybrid_clock.cc:648] HybridClock initialized: now 1785984541786028 us; error 53 us; skew 500 ppm
I20260806 02:49:01.786526 16761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:49:01 dist-test-slave-8tdl krb5kdc[15840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984541, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:01.790055 16761 init.cc:377] Logged in from keytab as kudu/127.1.210.67@KRBTEST.COM (short username kudu)
I20260806 02:49:01.791884 16761 webserver.cc:460] Webserver started at http://127.1.210.67:38321/ using document root <none> and password file <none>
I20260806 02:49:01.792677 16761 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:01.792838 16761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:01.798966 16761 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:49:01.802248 16777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:01.804016 16761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:49:01.804226 16761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/wal
uuid: "f24ecb82f42249ee97edfab14084c27c"
format_stamp: "Formatted at 2026-08-06 02:48:58 on dist-test-slave-8tdl"
server_key: "d7cd5a9d507b9dda6592ef516e507b05"
server_key_iv: "4b65943704d813758b31e6d394e0a541"
server_key_version: "encryptionkey@0"
I20260806 02:49:01.805600 16761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:01.820356 16761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:01.825415 16761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:01.825711 16761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:01.826817 16761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:49:01.828306 16761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:49:01.828433 16761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:01.828562 16761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:49:01.828646 16761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:01.865046 16761 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.67:44225
I20260806 02:49:01.865164 16889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.67:44225 every 8 connection(s)
I20260806 02:49:01.866739 16761 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
I20260806 02:49:01.868534 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 16761
Aug 06 02:49:01 dist-test-slave-8tdl krb5kdc[15840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984541, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:49:01.885733 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:01.869673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35678 (local address 127.1.210.126:45859)
0806 02:49:01.869863 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:01.869870 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:01.870636 (+ 766us) server_negotiation.cc:407] Connection header received
0806 02:49:01.870903 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:01.870913 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:01.870982 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:01.871095 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:01.872709 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:01.873501 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:01.875628 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:01.876376 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:01.876521 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:01.879423 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:01.879429 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:01.879432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:01.879468 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:01.882475 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:01.883263 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:01.883268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:01.883272 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:01.883486 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:01.884044 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:01.884051 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:01.884056 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:01.884490 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:01.884785 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:01.885043 (+ 258us) server_negotiation.cc:299] Negotiation successful
0806 02:49:01.885325 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:49:01.890494 16893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:01.869810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:46227 (local address 127.1.210.126:45859)
0806 02:49:01.870312 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:01.870319 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:01.871644 (+ 1325us) server_negotiation.cc:407] Connection header received
0806 02:49:01.873527 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:01.873532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:01.873616 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:01.873715 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:01.875875 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:01.877020 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:01.878651 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:01.879156 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:01.879287 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:01.883217 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:01.883222 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:01.883225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:01.883258 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:01.886887 (+ 3629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:01.887912 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:01.887916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:01.887918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:01.888052 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:01.888612 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:01.888618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:01.888621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:01.888993 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:01.889161 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:01.889985 (+ 824us) server_negotiation.cc:299] Negotiation successful
0806 02:49:01.890167 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":263,"threads_started":1}
I20260806 02:49:01.891697 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:01.870571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.67:46227)
0806 02:49:01.871468 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:01.871520 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:01.873275 (+ 1755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:01.874018 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:01.874030 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:01.874818 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:49:01.875693 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:01.875716 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:01.877193 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:01.877197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:01.878492 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:01.878506 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:01.879407 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:01.879415 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:01.879655 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:01.880387 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:49:01.880411 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:49:01.882974 (+ 2563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:49:01.887072 (+ 4098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:01.887081 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:01.887097 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:01.887777 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:01.888183 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:01.888188 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:01.888191 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:01.888464 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:01.889210 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:49:01.889217 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:49:01.889603 (+ 386us) client_negotiation.cc:769] Sending connection context
0806 02:49:01.889868 (+ 265us) client_negotiation.cc:240] Negotiation successful
0806 02:49:01.890458 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":365,"threads_started":1}
I20260806 02:49:01.893546 16890 heartbeater.cc:347] Connected to a master server at 127.1.210.126:45859
I20260806 02:49:01.893831 16890 heartbeater.cc:464] Registering TS with master...
I20260806 02:49:01.894616 16890 heartbeater.cc:511] Master 127.1.210.126:45859 requested a full tablet report, sending...
I20260806 02:49:01.896273 16434 ts_manager.cc:194] Registered new tserver with Master: f24ecb82f42249ee97edfab14084c27c (127.1.210.67:44225)
I20260806 02:49:01.897166 16434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.67@KRBTEST.COM'} at 127.1.210.67:46227
I20260806 02:49:01.901953 1865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:49:01.910142 16893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:01.903148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35686 (local address 127.1.210.126:45859)
0806 02:49:01.903387 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:01.903394 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:01.903742 (+ 348us) server_negotiation.cc:407] Connection header received
0806 02:49:01.903928 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:01.903932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:01.904086 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:01.904215 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:49:01.904751 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:01.905570 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:01.907487 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:01.907951 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:01.908070 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:01.908647 (+ 577us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:49:01.909259 (+ 612us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:49:01.909377 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:01.909582 (+ 205us) server_negotiation.cc:299] Negotiation successful
0806 02:49:01.909669 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 02:49:01.911535 16434 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:35686:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:49:01.919775 16434 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:49:01.939981 16622 heartbeater.cc:503] Master 127.1.210.126:45859 was elected leader, sending a full tablet report...
I20260806 02:49:01.950770 16901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:01.934791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:44225 (local address 127.0.0.1:50814)
0806 02:49:01.935849 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:01.935905 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:01.936095 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:01.939185 (+ 3090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:01.939196 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:01.939242 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:01.939772 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:01.939788 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:01.943316 (+ 3528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:01.943325 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:01.946655 (+ 3330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:01.946708 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:01.949395 (+ 2687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:01.949411 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:01.949761 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:01.949838 (+ 77us) client_negotiation.cc:769] Sending connection context
0806 02:49:01.949982 (+ 144us) client_negotiation.cc:240] Negotiation successful
0806 02:49:01.950241 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"thread_start_us":541,"threads_started":1}
I20260806 02:49:01.951308 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:01.935080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50814 (local address 127.1.210.67:44225)
0806 02:49:01.936829 (+ 1749us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:01.936839 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:01.936871 (+ 32us) server_negotiation.cc:407] Connection header received
0806 02:49:01.936993 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:01.937005 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:01.937357 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:01.937579 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:01.940287 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:01.943106 (+ 2819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:01.947007 (+ 3901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:01.949169 (+ 2162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:01.949325 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:01.949591 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:01.950353 (+ 762us) server_negotiation.cc:299] Negotiation successful
0806 02:49:01.950567 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1546,"thread_start_us":1657,"threads_started":1}
I20260806 02:49:01.955808 16825 tablet_service.cc:1467] Processing CreateTablet for tablet bee5347946c0455b9e352bd65446e095 (DEFAULT_TABLE table=test-table [id=fac5712a056b480facf4366d58186051]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:49:01.957281 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:01.936308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:45613 (local address 127.0.0.1:54146)
0806 02:49:01.947519 (+ 11211us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:01.947539 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:01.947674 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:01.950006 (+ 2332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:01.950010 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:01.950039 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:01.950495 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:01.950509 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:01.952382 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:01.952390 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:01.954356 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:01.954372 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:01.956358 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:01.956365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:01.956568 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:01.956605 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 02:49:01.956710 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 02:49:01.956864 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11065,"mutex_wait_us":80,"thread_start_us":12354,"threads_started":1}
I20260806 02:49:01.957589 16825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bee5347946c0455b9e352bd65446e095. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:01.957331 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:01.936584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54146 (local address 127.1.210.66:45613)
0806 02:49:01.949336 (+ 12752us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:01.949347 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:01.949373 (+ 26us) server_negotiation.cc:407] Connection header received
0806 02:49:01.949471 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:01.949480 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:01.949772 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:01.950008 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:01.950758 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:01.952194 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:01.954561 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:01.956110 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:01.956267 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:01.956469 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:01.956797 (+ 328us) server_negotiation.cc:299] Negotiation successful
0806 02:49:01.956983 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12546,"thread_start_us":12388,"threads_started":1}
I20260806 02:49:01.958437 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:01.936773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:35323 (local address 127.0.0.1:55882)
0806 02:49:01.949005 (+ 12232us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:01.949027 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:01.949183 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:01.950305 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:01.950310 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:01.950338 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:01.950889 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:01.950904 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:01.953396 (+ 2492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:01.953406 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:01.955501 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:01.955514 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:01.957598 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:01.957604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:01.957782 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:01.957821 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 02:49:01.957929 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 02:49:01.958072 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12081,"mutex_wait_us":105,"thread_start_us":11339,"threads_started":1}
I20260806 02:49:01.958650 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:01.937673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55882 (local address 127.1.210.65:35323)
0806 02:49:01.945505 (+ 7832us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:01.945520 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:01.949344 (+ 3824us) server_negotiation.cc:407] Connection header received
0806 02:49:01.949472 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:01.949480 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:01.949777 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:01.950007 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:01.951129 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:01.953219 (+ 2090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:01.955674 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:01.957011 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:01.957165 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:01.957423 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:01.958020 (+ 597us) server_negotiation.cc:299] Negotiation successful
0806 02:49:01.958203 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7559,"thread_start_us":7342,"threads_started":1}
I20260806 02:49:01.962307 16692 tablet_service.cc:1467] Processing CreateTablet for tablet bee5347946c0455b9e352bd65446e095 (DEFAULT_TABLE table=test-table [id=fac5712a056b480facf4366d58186051]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:49:01.962618 16556 tablet_service.cc:1467] Processing CreateTablet for tablet bee5347946c0455b9e352bd65446e095 (DEFAULT_TABLE table=test-table [id=fac5712a056b480facf4366d58186051]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:49:01.963718 16692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bee5347946c0455b9e352bd65446e095. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:01.964119 16556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bee5347946c0455b9e352bd65446e095. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:01.974696 16907 tablet_bootstrap.cc:492] T bee5347946c0455b9e352bd65446e095 P f24ecb82f42249ee97edfab14084c27c: Bootstrap starting.
I20260806 02:49:01.977381 16908 tablet_bootstrap.cc:492] T bee5347946c0455b9e352bd65446e095 P 88480c29dea44647ba7cc9e90c927baa: Bootstrap starting.
I20260806 02:49:01.979777 16907 tablet_bootstrap.cc:654] T bee5347946c0455b9e352bd65446e095 P f24ecb82f42249ee97edfab14084c27c: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:01.980589 16909 tablet_bootstrap.cc:492] T bee5347946c0455b9e352bd65446e095 P a2a25e0aaf014da8ab4bac97925b34d5: Bootstrap starting.
I20260806 02:49:01.981228 16908 tablet_bootstrap.cc:654] T bee5347946c0455b9e352bd65446e095 P 88480c29dea44647ba7cc9e90c927baa: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:01.981472 16907 log.cc:824] T bee5347946c0455b9e352bd65446e095 P f24ecb82f42249ee97edfab14084c27c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:01.982483 16908 log.cc:824] T bee5347946c0455b9e352bd65446e095 P 88480c29dea44647ba7cc9e90c927baa: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:01.984752 16909 tablet_bootstrap.cc:654] T bee5347946c0455b9e352bd65446e095 P a2a25e0aaf014da8ab4bac97925b34d5: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:01.985214 16907 tablet_bootstrap.cc:492] T bee5347946c0455b9e352bd65446e095 P f24ecb82f42249ee97edfab14084c27c: No bootstrap required, opened a new log
I20260806 02:49:01.985527 16907 ts_tablet_manager.cc:1397] T bee5347946c0455b9e352bd65446e095 P f24ecb82f42249ee97edfab14084c27c: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260806 02:49:01.986068 16909 log.cc:824] T bee5347946c0455b9e352bd65446e095 P a2a25e0aaf014da8ab4bac97925b34d5: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:01.989679 16909 tablet_bootstrap.cc:492] T bee5347946c0455b9e352bd65446e095 P a2a25e0aaf014da8ab4bac97925b34d5: No bootstrap required, opened a new log
I20260806 02:49:01.989686 16908 tablet_bootstrap.cc:492] T bee5347946c0455b9e352bd65446e095 P 88480c29dea44647ba7cc9e90c927baa: No bootstrap required, opened a new log
I20260806 02:49:01.989988 16908 ts_tablet_manager.cc:1397] T bee5347946c0455b9e352bd65446e095 P 88480c29dea44647ba7cc9e90c927baa: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260806 02:49:01.989989 16909 ts_tablet_manager.cc:1397] T bee5347946c0455b9e352bd65446e095 P a2a25e0aaf014da8ab4bac97925b34d5: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260806 02:49:01.994028 16907 raft_consensus.cc:348] T bee5347946c0455b9e352bd65446e095 P f24ecb82f42249ee97edfab14084c27c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24ecb82f42249ee97edfab14084c27c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44225 } } peers { permanent_uuid: "88480c29dea44647ba7cc9e90c927baa" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45613 } } peers { permanent_uuid: "a2a25e0aaf014da8ab4bac97925b34d5" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 35323 } }
I20260806 02:49:01.994445 16907 raft_consensus.cc:374] T bee5347946c0455b9e352bd65446e095 P f24ecb82f42249ee97edfab14084c27c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:01.994544 16907 raft_consensus.cc:729] T bee5347946c0455b9e352bd65446e095 P f24ecb82f42249ee97edfab14084c27c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24ecb82f42249ee97edfab14084c27c, State: Initialized, Role: FOLLOWER
I20260806 02:49:01.995270 16907 consensus_queue.cc:260] T bee5347946c0455b9e352bd65446e095 P f24ecb82f42249ee97edfab14084c27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24ecb82f42249ee97edfab14084c27c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44225 } } peers { permanent_uuid: "88480c29dea44647ba7cc9e90c927baa" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45613 } } peers { permanent_uuid: "a2a25e0aaf014da8ab4bac97925b34d5" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 35323 } }
I20260806 02:49:01.996436 16890 heartbeater.cc:503] Master 127.1.210.126:45859 was elected leader, sending a full tablet report...
I20260806 02:49:01.997164 16907 ts_tablet_manager.cc:1428] T bee5347946c0455b9e352bd65446e095 P f24ecb82f42249ee97edfab14084c27c: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260806 02:49:02.001226 16908 raft_consensus.cc:348] T bee5347946c0455b9e352bd65446e095 P 88480c29dea44647ba7cc9e90c927baa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24ecb82f42249ee97edfab14084c27c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44225 } } peers { permanent_uuid: "88480c29dea44647ba7cc9e90c927baa" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45613 } } peers { permanent_uuid: "a2a25e0aaf014da8ab4bac97925b34d5" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 35323 } }
I20260806 02:49:02.001226 16909 raft_consensus.cc:348] T bee5347946c0455b9e352bd65446e095 P a2a25e0aaf014da8ab4bac97925b34d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24ecb82f42249ee97edfab14084c27c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44225 } } peers { permanent_uuid: "88480c29dea44647ba7cc9e90c927baa" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45613 } } peers { permanent_uuid: "a2a25e0aaf014da8ab4bac97925b34d5" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 35323 } }
I20260806 02:49:02.001758 16909 raft_consensus.cc:374] T bee5347946c0455b9e352bd65446e095 P a2a25e0aaf014da8ab4bac97925b34d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:02.001758 16908 raft_consensus.cc:374] T bee5347946c0455b9e352bd65446e095 P 88480c29dea44647ba7cc9e90c927baa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:02.001883 16909 raft_consensus.cc:729] T bee5347946c0455b9e352bd65446e095 P a2a25e0aaf014da8ab4bac97925b34d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a25e0aaf014da8ab4bac97925b34d5, State: Initialized, Role: FOLLOWER
I20260806 02:49:02.001875 16908 raft_consensus.cc:729] T bee5347946c0455b9e352bd65446e095 P 88480c29dea44647ba7cc9e90c927baa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88480c29dea44647ba7cc9e90c927baa, State: Initialized, Role: FOLLOWER
I20260806 02:49:02.002574 16908 consensus_queue.cc:260] T bee5347946c0455b9e352bd65446e095 P 88480c29dea44647ba7cc9e90c927baa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24ecb82f42249ee97edfab14084c27c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44225 } } peers { permanent_uuid: "88480c29dea44647ba7cc9e90c927baa" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45613 } } peers { permanent_uuid: "a2a25e0aaf014da8ab4bac97925b34d5" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 35323 } }
I20260806 02:49:02.002558 16909 consensus_queue.cc:260] T bee5347946c0455b9e352bd65446e095 P a2a25e0aaf014da8ab4bac97925b34d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24ecb82f42249ee97edfab14084c27c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44225 } } peers { permanent_uuid: "88480c29dea44647ba7cc9e90c927baa" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45613 } } peers { permanent_uuid: "a2a25e0aaf014da8ab4bac97925b34d5" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 35323 } }
I20260806 02:49:02.004420 16757 heartbeater.cc:503] Master 127.1.210.126:45859 was elected leader, sending a full tablet report...
I20260806 02:49:02.005167 16909 ts_tablet_manager.cc:1428] T bee5347946c0455b9e352bd65446e095 P a2a25e0aaf014da8ab4bac97925b34d5: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260806 02:49:02.005354 16908 ts_tablet_manager.cc:1428] T bee5347946c0455b9e352bd65446e095 P 88480c29dea44647ba7cc9e90c927baa: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260806 02:49:02.012923 16913 raft_consensus.cc:482] T bee5347946c0455b9e352bd65446e095 P f24ecb82f42249ee97edfab14084c27c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:49:02.013262 16913 raft_consensus.cc:504] T bee5347946c0455b9e352bd65446e095 P f24ecb82f42249ee97edfab14084c27c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24ecb82f42249ee97edfab14084c27c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44225 } } peers { permanent_uuid: "88480c29dea44647ba7cc9e90c927baa" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45613 } } peers { permanent_uuid: "a2a25e0aaf014da8ab4bac97925b34d5" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 35323 } }
I20260806 02:49:02.015228 16913 leader_election.cc:288] T bee5347946c0455b9e352bd65446e095 P f24ecb82f42249ee97edfab14084c27c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88480c29dea44647ba7cc9e90c927baa (127.1.210.66:45613), a2a25e0aaf014da8ab4bac97925b34d5 (127.1.210.65:35323)
I20260806 02:49:02.021682 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:02.015710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:55097 (local address 127.1.210.66:45613)
0806 02:49:02.016386 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:02.016393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:02.016418 (+ 25us) server_negotiation.cc:407] Connection header received
0806 02:49:02.016499 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:02.016508 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:02.016601 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:02.016862 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:02.017436 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:02.018159 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:02.019933 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:02.020726 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:02.020902 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:02.021000 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:02.021286 (+ 286us) server_negotiation.cc:299] Negotiation successful
0806 02:49:02.021412 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468}
I20260806 02:49:02.021690 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:02.015456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:45613 (local address 127.1.210.67:55097)
0806 02:49:02.015804 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:02.015827 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:02.015985 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:02.017020 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:02.017026 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:02.017060 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:02.017269 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:02.017285 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:02.018306 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:02.018312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:02.019761 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:02.019775 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:02.020898 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:02.020904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:02.021109 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:02.021133 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 02:49:02.021241 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 02:49:02.021381 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197}
I20260806 02:49:02.022706 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:02.016076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:46385 (local address 127.1.210.65:35323)
0806 02:49:02.016618 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:02.016625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:02.017230 (+ 605us) server_negotiation.cc:407] Connection header received
0806 02:49:02.017456 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:02.017460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:02.017521 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:02.017623 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:02.018174 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:02.018982 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:02.020668 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:02.021613 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:02.021800 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:02.021877 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:02.022287 (+ 410us) server_negotiation.cc:299] Negotiation successful
0806 02:49:02.022434 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380}
I20260806 02:49:02.022799 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:02.015773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:35323 (local address 127.1.210.67:46385)
0806 02:49:02.017133 (+ 1360us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:02.017154 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:02.017291 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:02.017643 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:02.017647 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:02.017673 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:02.017991 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:02.017999 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:02.019135 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:02.019141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:02.020501 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:02.020512 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:02.021823 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:02.021830 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:02.022038 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:02.022076 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 02:49:02.022192 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 02:49:02.022325 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1204,"thread_start_us":475,"threads_started":1}
I20260806 02:49:02.022910 16712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee5347946c0455b9e352bd65446e095" candidate_uuid: "f24ecb82f42249ee97edfab14084c27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88480c29dea44647ba7cc9e90c927baa" is_pre_election: true
I20260806 02:49:02.023525 16712 raft_consensus.cc:2454] T bee5347946c0455b9e352bd65446e095 P 88480c29dea44647ba7cc9e90c927baa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24ecb82f42249ee97edfab14084c27c in term 0.
I20260806 02:49:02.023885 16577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee5347946c0455b9e352bd65446e095" candidate_uuid: "f24ecb82f42249ee97edfab14084c27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a25e0aaf014da8ab4bac97925b34d5" is_pre_election: true
I20260806 02:49:02.024432 16779 leader_election.cc:302] T bee5347946c0455b9e352bd65446e095 P f24ecb82f42249ee97edfab14084c27c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88480c29dea44647ba7cc9e90c927baa, f24ecb82f42249ee97edfab14084c27c; no voters:
I20260806 02:49:02.024571 16577 raft_consensus.cc:2454] T bee5347946c0455b9e352bd65446e095 P a2a25e0aaf014da8ab4bac97925b34d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f24ecb82f42249ee97edfab14084c27c in term 0.
I20260806 02:49:02.024958 16913 raft_consensus.cc:2781] T bee5347946c0455b9e352bd65446e095 P f24ecb82f42249ee97edfab14084c27c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:49:02.025126 16913 raft_consensus.cc:482] T bee5347946c0455b9e352bd65446e095 P f24ecb82f42249ee97edfab14084c27c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:49:02.025264 16913 raft_consensus.cc:3037] T bee5347946c0455b9e352bd65446e095 P f24ecb82f42249ee97edfab14084c27c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:02.027727 16913 raft_consensus.cc:504] T bee5347946c0455b9e352bd65446e095 P f24ecb82f42249ee97edfab14084c27c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24ecb82f42249ee97edfab14084c27c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44225 } } peers { permanent_uuid: "88480c29dea44647ba7cc9e90c927baa" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45613 } } peers { permanent_uuid: "a2a25e0aaf014da8ab4bac97925b34d5" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 35323 } }
I20260806 02:49:02.028476 16913 leader_election.cc:288] T bee5347946c0455b9e352bd65446e095 P f24ecb82f42249ee97edfab14084c27c [CANDIDATE]: Term 1 election: Requested vote from peers 88480c29dea44647ba7cc9e90c927baa (127.1.210.66:45613), a2a25e0aaf014da8ab4bac97925b34d5 (127.1.210.65:35323)
I20260806 02:49:02.029178 16577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee5347946c0455b9e352bd65446e095" candidate_uuid: "f24ecb82f42249ee97edfab14084c27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a25e0aaf014da8ab4bac97925b34d5"
I20260806 02:49:02.029140 16712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee5347946c0455b9e352bd65446e095" candidate_uuid: "f24ecb82f42249ee97edfab14084c27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88480c29dea44647ba7cc9e90c927baa"
I20260806 02:49:02.029414 16712 raft_consensus.cc:3037] T bee5347946c0455b9e352bd65446e095 P 88480c29dea44647ba7cc9e90c927baa [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:02.029429 16577 raft_consensus.cc:3037] T bee5347946c0455b9e352bd65446e095 P a2a25e0aaf014da8ab4bac97925b34d5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:02.032954 16712 raft_consensus.cc:2454] T bee5347946c0455b9e352bd65446e095 P 88480c29dea44647ba7cc9e90c927baa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24ecb82f42249ee97edfab14084c27c in term 1.
I20260806 02:49:02.032954 16577 raft_consensus.cc:2454] T bee5347946c0455b9e352bd65446e095 P a2a25e0aaf014da8ab4bac97925b34d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f24ecb82f42249ee97edfab14084c27c in term 1.
I20260806 02:49:02.033538 16780 leader_election.cc:302] T bee5347946c0455b9e352bd65446e095 P f24ecb82f42249ee97edfab14084c27c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a25e0aaf014da8ab4bac97925b34d5, f24ecb82f42249ee97edfab14084c27c; no voters:
I20260806 02:49:02.033918 16913 raft_consensus.cc:2781] T bee5347946c0455b9e352bd65446e095 P f24ecb82f42249ee97edfab14084c27c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:49:02.034437 16913 raft_consensus.cc:686] T bee5347946c0455b9e352bd65446e095 P f24ecb82f42249ee97edfab14084c27c [term 1 LEADER]: Becoming Leader. State: Replica: f24ecb82f42249ee97edfab14084c27c, State: Running, Role: LEADER
I20260806 02:49:02.035015 16913 consensus_queue.cc:237] T bee5347946c0455b9e352bd65446e095 P f24ecb82f42249ee97edfab14084c27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24ecb82f42249ee97edfab14084c27c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44225 } } peers { permanent_uuid: "88480c29dea44647ba7cc9e90c927baa" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45613 } } peers { permanent_uuid: "a2a25e0aaf014da8ab4bac97925b34d5" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 35323 } }
I20260806 02:49:02.040457 16433 catalog_manager.cc:5495] T bee5347946c0455b9e352bd65446e095 P f24ecb82f42249ee97edfab14084c27c reported cstate change: term changed from 0 to 1, leader changed from <none> to f24ecb82f42249ee97edfab14084c27c (127.1.210.67). New cstate: current_term: 1 leader_uuid: "f24ecb82f42249ee97edfab14084c27c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f24ecb82f42249ee97edfab14084c27c" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88480c29dea44647ba7cc9e90c927baa" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a25e0aaf014da8ab4bac97925b34d5" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 35323 } health_report { overall_health: UNKNOWN } } }
I20260806 02:49:02.076153 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:02.070128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50824 (local address 127.1.210.67:44225)
0806 02:49:02.070341 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:02.070346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:02.070364 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:49:02.070491 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:02.070496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:02.070597 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:02.070806 (+ 209us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:49:02.071505 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:02.072325 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:02.073752 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:02.074205 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:02.074316 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:02.074949 (+ 633us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:49:02.075564 (+ 615us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:49:02.075684 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:02.075819 (+ 135us) server_negotiation.cc:299] Negotiation successful
0806 02:49:02.075895 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 02:49:02.085388 16577 raft_consensus.cc:1261] T bee5347946c0455b9e352bd65446e095 P a2a25e0aaf014da8ab4bac97925b34d5 [term 1 FOLLOWER]: Refusing update from remote peer f24ecb82f42249ee97edfab14084c27c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:49:02.085388 16712 raft_consensus.cc:1261] T bee5347946c0455b9e352bd65446e095 P 88480c29dea44647ba7cc9e90c927baa [term 1 FOLLOWER]: Refusing update from remote peer f24ecb82f42249ee97edfab14084c27c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:49:02.086696 16913 consensus_queue.cc:1035] T bee5347946c0455b9e352bd65446e095 P f24ecb82f42249ee97edfab14084c27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "88480c29dea44647ba7cc9e90c927baa" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 45613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:49:02.087145 16917 consensus_queue.cc:1035] T bee5347946c0455b9e352bd65446e095 P f24ecb82f42249ee97edfab14084c27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a25e0aaf014da8ab4bac97925b34d5" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:49:02.101511 16922 mvcc.cc:205] Tried to move back new op lower bound from 7315392684368912384 to 7315392684179935232. Current Snapshot: MvccSnapshot[applied={T|T < 7315392684368912384}]
I20260806 02:49:02.114543 16434 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:35686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:49:02.114921 16434 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:35686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:49:02.119171 16434 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80: Sending DeleteTablet for 3 replicas of tablet bee5347946c0455b9e352bd65446e095
I20260806 02:49:02.120723 16825 tablet_service.cc:1514] Processing DeleteTablet for tablet bee5347946c0455b9e352bd65446e095 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:49:02 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:50814
I20260806 02:49:02.120729 16556 tablet_service.cc:1514] Processing DeleteTablet for tablet bee5347946c0455b9e352bd65446e095 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:49:02 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:55882
I20260806 02:49:02.121476 16692 tablet_service.cc:1514] Processing DeleteTablet for tablet bee5347946c0455b9e352bd65446e095 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:49:02 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:54146
I20260806 02:49:02.124770 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 16472
I20260806 02:49:02.133853 16933 tablet_replica.cc:330] T bee5347946c0455b9e352bd65446e095 P 88480c29dea44647ba7cc9e90c927baa: stopping tablet replica
I20260806 02:49:02.134459 16933 raft_consensus.cc:2229] T bee5347946c0455b9e352bd65446e095 P 88480c29dea44647ba7cc9e90c927baa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:49:02.134899 16931 tablet_replica.cc:330] T bee5347946c0455b9e352bd65446e095 P f24ecb82f42249ee97edfab14084c27c: stopping tablet replica
I20260806 02:49:02.135106 16933 raft_consensus.cc:2258] T bee5347946c0455b9e352bd65446e095 P 88480c29dea44647ba7cc9e90c927baa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:49:02.135756 16931 raft_consensus.cc:2229] T bee5347946c0455b9e352bd65446e095 P f24ecb82f42249ee97edfab14084c27c [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:49:02.136722 16931 raft_consensus.cc:2258] T bee5347946c0455b9e352bd65446e095 P f24ecb82f42249ee97edfab14084c27c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:49:02.137475 16932 tablet_replica.cc:330] T bee5347946c0455b9e352bd65446e095 P a2a25e0aaf014da8ab4bac97925b34d5: stopping tablet replica
I20260806 02:49:02.137945 16932 raft_consensus.cc:2229] T bee5347946c0455b9e352bd65446e095 P a2a25e0aaf014da8ab4bac97925b34d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:49:02.138777 16932 raft_consensus.cc:2258] T bee5347946c0455b9e352bd65446e095 P a2a25e0aaf014da8ab4bac97925b34d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:49:02.139789 16931 ts_tablet_manager.cc:1905] T bee5347946c0455b9e352bd65446e095 P f24ecb82f42249ee97edfab14084c27c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:49:02.141067 16933 ts_tablet_manager.cc:1905] T bee5347946c0455b9e352bd65446e095 P 88480c29dea44647ba7cc9e90c927baa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:49:02.141223 16932 ts_tablet_manager.cc:1905] T bee5347946c0455b9e352bd65446e095 P a2a25e0aaf014da8ab4bac97925b34d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:49:02.146226 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:02.125924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55888 (local address 127.1.210.65:35323)
0806 02:49:02.126161 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:02.126170 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:02.126234 (+ 64us) server_negotiation.cc:407] Connection header received
0806 02:49:02.126455 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:02.126463 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:02.126598 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:02.126802 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:02.128855 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:02.130474 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:02.132795 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:02.133664 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:02.133811 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:02.137934 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:02.137953 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:02.137973 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:02.138025 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:02.142503 (+ 4478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:02.143611 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:02.143622 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:02.143625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:02.143814 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:02.144318 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:02.144323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:02.144327 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:02.144854 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:02.145199 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:02.145513 (+ 314us) server_negotiation.cc:299] Negotiation successful
0806 02:49:02.145784 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 02:49:02.147217 16617 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:49:02.148342 16931 ts_tablet_manager.cc:1918] T bee5347946c0455b9e352bd65446e095 P f24ecb82f42249ee97edfab14084c27c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:49:02.148684 16931 log.cc:1196] T bee5347946c0455b9e352bd65446e095 P f24ecb82f42249ee97edfab14084c27c: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/wal/wals/bee5347946c0455b9e352bd65446e095
I20260806 02:49:02.149544 16931 ts_tablet_manager.cc:1939] T bee5347946c0455b9e352bd65446e095 P f24ecb82f42249ee97edfab14084c27c: Deleting consensus metadata
I20260806 02:49:02.152050 16933 ts_tablet_manager.cc:1918] T bee5347946c0455b9e352bd65446e095 P 88480c29dea44647ba7cc9e90c927baa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:49:02.152319 16933 log.cc:1196] T bee5347946c0455b9e352bd65446e095 P 88480c29dea44647ba7cc9e90c927baa: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/wal/wals/bee5347946c0455b9e352bd65446e095
I20260806 02:49:02.152671 16419 catalog_manager.cc:4841] TS f24ecb82f42249ee97edfab14084c27c (127.1.210.67:44225): tablet bee5347946c0455b9e352bd65446e095 (table test-table [id=fac5712a056b480facf4366d58186051]) successfully deleted
I20260806 02:49:02.153067 16933 ts_tablet_manager.cc:1939] T bee5347946c0455b9e352bd65446e095 P 88480c29dea44647ba7cc9e90c927baa: Deleting consensus metadata
I20260806 02:49:02.155529 16419 catalog_manager.cc:4841] TS 88480c29dea44647ba7cc9e90c927baa (127.1.210.66:45613): tablet bee5347946c0455b9e352bd65446e095 (table test-table [id=fac5712a056b480facf4366d58186051]) successfully deleted
I20260806 02:49:02.251453 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 16472
W20260806 02:49:02.273774 16420 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.210.65:35323 (error 108)
I20260806 02:49:02.274586 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 16626
I20260806 02:49:02.276229 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:02.275465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:35323 (local address 127.0.0.1:55894)
0806 02:49:02.275899 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:02.276042 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:35323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309}
W20260806 02:49:02.276631 16420 catalog_manager.cc:4584] TS a2a25e0aaf014da8ab4bac97925b34d5 (127.1.210.65:35323): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bee5347946c0455b9e352bd65446e095: Network error: Client connection negotiation failed: client connection to 127.1.210.65:35323: connect: Connection refused (error 111)
I20260806 02:49:02.291973 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:02.275560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54150 (local address 127.1.210.66:45613)
0806 02:49:02.275753 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:02.275757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:02.275908 (+ 151us) server_negotiation.cc:407] Connection header received
0806 02:49:02.276112 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:02.276118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:02.276203 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:02.276327 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:02.278168 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:02.278976 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:02.280942 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:02.281377 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:02.281470 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:02.284591 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:02.284614 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:02.284636 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:02.284698 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:02.288857 (+ 4159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:02.289665 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:02.289677 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:02.289681 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:02.289850 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:02.290236 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:02.290241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:02.290245 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:02.290807 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:02.291124 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:02.291356 (+ 232us) server_negotiation.cc:299] Negotiation successful
0806 02:49:02.291590 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 02:49:02.292806 16752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:49:02.330720 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:02.330109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:35323 (local address 127.0.0.1:55910)
0806 02:49:02.330346 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:02.330484 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:35323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260806 02:49:02.331970 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:02.331513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:35323 (local address 127.0.0.1:55924)
0806 02:49:02.331695 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:02.331809 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:35323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260806 02:49:02.393071 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:02.392411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:35323 (local address 127.0.0.1:55936)
0806 02:49:02.392725 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:02.392844 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:35323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260806 02:49:02.394503 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:02.394008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:35323 (local address 127.0.0.1:55942)
0806 02:49:02.394219 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:02.394336 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:35323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260806 02:49:02.397501 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 16626
I20260806 02:49:02.424860 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 16761
I20260806 02:49:02.441813 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:02.425775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50826 (local address 127.1.210.67:44225)
0806 02:49:02.425981 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:02.425989 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:02.426028 (+ 39us) server_negotiation.cc:407] Connection header received
0806 02:49:02.426285 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:02.426291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:02.426379 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:02.426500 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:02.427993 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:02.428950 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:02.430530 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:02.431222 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:02.431385 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:02.434340 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:02.434354 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:02.434370 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:02.434427 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:02.438166 (+ 3739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:02.439246 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:02.439255 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:02.439258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:02.439485 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:02.439951 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:02.439959 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:02.439963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:02.440363 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:02.440773 (+ 410us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:02.441119 (+ 346us) server_negotiation.cc:299] Negotiation successful
0806 02:49:02.441378 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 02:49:02.442835 16885 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:49:02.475667 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:02.474904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:35323 (local address 127.0.0.1:55958)
0806 02:49:02.475123 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:02.475243 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:35323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260806 02:49:02.476890 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:02.476465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:35323 (local address 127.0.0.1:55964)
0806 02:49:02.476671 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:02.476762 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:35323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260806 02:49:02.574885 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 16761
I20260806 02:49:02.600097 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 16401
I20260806 02:49:02.601526 16465 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:49:02.724431 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:02.723781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:35323 (local address 127.0.0.1:55968)
0806 02:49:02.724050 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:02.724203 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:35323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260806 02:49:02.724476 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 16401
I20260806 02:49:02.751870 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:45859
--webserver_interface=127.1.210.126
--webserver_port=35143
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:45859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:49:03.113209 16939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:03.113528 16939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:03.113622 16939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:03.121318 16939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:49:03.121469 16939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:03.121521 16939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:03.121587 16939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:49:03.121636 16939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:49:03.132277 16939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:45859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:45859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=35143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:49:03.133701 16939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:03.135066 16939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:03.147426 16947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:03.147512 16944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:03.147486 16945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:03.149083 16939 server_base.cc:1034] running on GCE node
I20260806 02:49:03.149950 16939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:03.151660 16939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:03.152917 16939 hybrid_clock.cc:648] HybridClock initialized: now 1785984543152826 us; error 60 us; skew 500 ppm
I20260806 02:49:03.153345 16939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:49:03 dist-test-slave-8tdl krb5kdc[15840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984543, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:03.156756 16939 init.cc:377] Logged in from keytab as kudu/127.1.210.126@KRBTEST.COM (short username kudu)
I20260806 02:49:03.158458 16939 webserver.cc:460] Webserver started at http://127.1.210.126:35143/ using document root <none> and password file <none>
I20260806 02:49:03.159282 16939 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:03.159439 16939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:03.165130 16939 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:49:03.168241 16954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:03.169708 16939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:49:03.169916 16939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "1b645c239c754d49a17f96a216c4af80"
format_stamp: "Formatted at 2026-08-06 02:48:57 on dist-test-slave-8tdl"
server_key: "80246ef024936cd5ac4eb7aed35dc814"
server_key_iv: "267a2538434db8f71d260a2b8d04f70e"
server_key_version: "encryptionkey@0"
I20260806 02:49:03.171103 16939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:03.190181 16939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:03.194568 16939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:03.194908 16939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:03.215515 16939 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:45859
I20260806 02:49:03.215534 17005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:45859 every 8 connection(s)
I20260806 02:49:03.217093 16939 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:49:03.221261 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 16939
I20260806 02:49:03.222302 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.65:35323
--local_ip_for_outbound_sockets=127.1.210.65
--tserver_master_addrs=127.1.210.126:45859
--webserver_port=44417
--webserver_interface=127.1.210.65
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:49:03.225721 17006 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:49:03.231199 17006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80: Bootstrap starting.
I20260806 02:49:03.253786 17006 log.cc:824] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:03.270151 17006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:49:03.270864 17006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80: Bootstrap complete.
I20260806 02:49:03.280758 17006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b645c239c754d49a17f96a216c4af80" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45859 } }
I20260806 02:49:03.281719 17006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b645c239c754d49a17f96a216c4af80, State: Initialized, Role: FOLLOWER
I20260806 02:49:03.282333 17006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b645c239c754d49a17f96a216c4af80" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45859 } }
I20260806 02:49:03.282591 17006 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:49:03.282739 17006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:49:03.282938 17006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [term 2 FOLLOWER]: Advancing to term 3
I20260806 02:49:03.287096 17006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b645c239c754d49a17f96a216c4af80" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45859 } }
I20260806 02:49:03.287627 17006 leader_election.cc:302] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b645c239c754d49a17f96a216c4af80; no voters:
I20260806 02:49:03.288371 17006 leader_election.cc:288] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 02:49:03.288507 17010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 02:49:03.289810 17010 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [term 3 LEADER]: Becoming Leader. State: Replica: 1b645c239c754d49a17f96a216c4af80, State: Running, Role: LEADER
I20260806 02:49:03.290441 17010 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b645c239c754d49a17f96a216c4af80" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45859 } }
I20260806 02:49:03.290858 17006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:49:03.293151 17012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b645c239c754d49a17f96a216c4af80. Latest consensus state: current_term: 3 leader_uuid: "1b645c239c754d49a17f96a216c4af80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b645c239c754d49a17f96a216c4af80" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45859 } } }
I20260806 02:49:03.293210 17011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1b645c239c754d49a17f96a216c4af80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b645c239c754d49a17f96a216c4af80" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45859 } } }
I20260806 02:49:03.293555 17012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:03.293613 17011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b645c239c754d49a17f96a216c4af80 [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:03.301210 17016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:49:03.312464 17016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:49:03.314700 17016 catalog_manager.cc:1259] Loaded cluster ID: 67ed54e8ea694172866975565c9c0c1b
I20260806 02:49:03.314849 17016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260806 02:49:03.317816 17016 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f31641699ad google::LogMessage::Fail() at ??:0
@ 0x7f316416c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f31641694de google::LogMessage::Flush() at ??:0
@ 0x7f316416aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3179b9c512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f3179b99976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f3179c21610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f3179c21439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f3166db5345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3166db891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f3166db8799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f3166d93eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f316a06d6db start_thread at ??:0
@ 0x7f316300361f clone at ??:0
W20260806 02:49:03.639199 17008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:03.639559 17008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:03.639742 17008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:03.647702 17008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:03.647918 17008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:03.648108 17008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.65
I20260806 02:49:03.658854 17008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.65:35323
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.65
--webserver_port=44417
--tserver_master_addrs=127.1.210.126:45859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.65
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:49:03.660320 17008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:03.661880 17008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:03.675654 17048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:03.675693 17051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:03.675871 17049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:03.677124 17008 server_base.cc:1034] running on GCE node
I20260806 02:49:03.677820 17008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:03.679008 17008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:03.680269 17008 hybrid_clock.cc:648] HybridClock initialized: now 1785984543680221 us; error 60 us; skew 500 ppm
I20260806 02:49:03.680634 17008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:49:03 dist-test-slave-8tdl krb5kdc[15840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984543, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:03.684145 17008 init.cc:377] Logged in from keytab as kudu/127.1.210.65@KRBTEST.COM (short username kudu)
I20260806 02:49:03.685730 17008 webserver.cc:460] Webserver started at http://127.1.210.65:44417/ using document root <none> and password file <none>
I20260806 02:49:03.686452 17008 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:03.686564 17008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:03.693470 17008 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 02:49:03.696992 17058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:03.698822 17008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:49:03.699031 17008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/wal
uuid: "a2a25e0aaf014da8ab4bac97925b34d5"
format_stamp: "Formatted at 2026-08-06 02:48:57 on dist-test-slave-8tdl"
server_key: "8e9a6c4da6256d24c0a019d170b7fbe8"
server_key_iv: "d9f8cfd24fdf0d01605fb2199f920d70"
server_key_version: "encryptionkey@0"
I20260806 02:49:03.700138 17008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:03.720463 17008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:03.727491 17008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:03.727797 17008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:03.728987 17008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:49:03.731168 17066 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260806 02:49:03.737216 17066 ts_tablet_manager.cc:1852] T bee5347946c0455b9e352bd65446e095 P a2a25e0aaf014da8ab4bac97925b34d5: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260806 02:49:03.738333 17066 ts_tablet_manager.cc:1905] T bee5347946c0455b9e352bd65446e095 P a2a25e0aaf014da8ab4bac97925b34d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:49:03.746537 17066 ts_tablet_manager.cc:1918] T bee5347946c0455b9e352bd65446e095 P a2a25e0aaf014da8ab4bac97925b34d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:49:03.746809 17066 log.cc:1196] T bee5347946c0455b9e352bd65446e095 P a2a25e0aaf014da8ab4bac97925b34d5: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/wal/wals/bee5347946c0455b9e352bd65446e095
I20260806 02:49:03.747404 17066 ts_tablet_manager.cc:1939] T bee5347946c0455b9e352bd65446e095 P a2a25e0aaf014da8ab4bac97925b34d5: Deleting consensus metadata
I20260806 02:49:03.747852 17008 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260806 02:49:03.748149 17008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.017s user 0.001s sys 0.000s
I20260806 02:49:03.748301 17008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:49:03.748396 17008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:03.786463 17008 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.65:35323
I20260806 02:49:03.786727 17172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.65:35323 every 8 connection(s)
I20260806 02:49:03.788034 17008 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
I20260806 02:49:03.793460 17175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:03.791438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.65:46065)
0806 02:49:03.792240 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:03.792501 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":325,"threads_started":1}
W20260806 02:49:03.794149 17173 heartbeater.cc:644] Failed to heartbeat to 127.1.210.126:45859 (0 consecutive failures): Network error: Failed to ping master at 127.1.210.126:45859: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
I20260806 02:49:03.794817 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 17008
I20260806 02:49:03.795164 17175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:03.794817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.65:33771)
0806 02:49:03.794981 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:03.795058 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 02:49:03.795616 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.66:45613
--local_ip_for_outbound_sockets=127.1.210.66
--tserver_master_addrs=127.1.210.126:45859
--webserver_port=34855
--webserver_interface=127.1.210.66
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:49:03.796128 17175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:03.795734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.65:57031)
0806 02:49:03.795934 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:03.796005 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260806 02:49:03.796365 17173 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:49:04.155025 17176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:04.155447 17176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:04.155552 17176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:04.163409 17176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:04.163527 17176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:04.163646 17176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.66
I20260806 02:49:04.173934 17176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.66:45613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.66
--webserver_port=34855
--tserver_master_addrs=127.1.210.126:45859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.66
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:49:04.175323 17176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:04.176666 17176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:04.191561 17182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:04.191547 17183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:04.191596 17185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:04.192659 17176 server_base.cc:1034] running on GCE node
I20260806 02:49:04.193390 17176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:04.194432 17176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:04.195711 17176 hybrid_clock.cc:648] HybridClock initialized: now 1785984544195652 us; error 89 us; skew 500 ppm
I20260806 02:49:04.196201 17176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:49:04 dist-test-slave-8tdl krb5kdc[15840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984544, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:04.200078 17176 init.cc:377] Logged in from keytab as kudu/127.1.210.66@KRBTEST.COM (short username kudu)
I20260806 02:49:04.201881 17176 webserver.cc:460] Webserver started at http://127.1.210.66:34855/ using document root <none> and password file <none>
I20260806 02:49:04.202663 17176 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:04.202890 17176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:04.209060 17176 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260806 02:49:04.212126 17192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:04.213872 17176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:49:04.214097 17176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/wal
uuid: "88480c29dea44647ba7cc9e90c927baa"
format_stamp: "Formatted at 2026-08-06 02:48:58 on dist-test-slave-8tdl"
server_key: "c296c893bba7c99aa5b7ed43e32966e4"
server_key_iv: "ba3458ecfafe0798a0e8692548627204"
server_key_version: "encryptionkey@0"
I20260806 02:49:04.215253 17176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:04.228178 17176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:04.232851 17176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:04.233194 17176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:04.234228 17176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:49:04.235738 17176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:49:04.235877 17176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:04.236023 17176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:49:04.236114 17176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:04.273512 17176 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.66:45613
I20260806 02:49:04.273605 17306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.66:45613 every 8 connection(s)
I20260806 02:49:04.275184 17176 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
I20260806 02:49:04.280651 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:04.278619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.66:52271)
0806 02:49:04.279440 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:04.279687 (+ 247us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":294,"threads_started":1}
W20260806 02:49:04.281323 17307 heartbeater.cc:644] Failed to heartbeat to 127.1.210.126:45859 (0 consecutive failures): Network error: Failed to ping master at 127.1.210.126:45859: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
I20260806 02:49:04.282202 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:04.281826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.66:56579)
0806 02:49:04.282021 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:04.282091 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260806 02:49:04.283087 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:04.282763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.66:35973)
0806 02:49:04.282875 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:04.282977 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260806 02:49:04.283311 17307 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 02:49:04.284583 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 17176
I20260806 02:49:04.285241 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.67:44225
--local_ip_for_outbound_sockets=127.1.210.67
--tserver_master_addrs=127.1.210.126:45859
--webserver_port=38321
--webserver_interface=127.1.210.67
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:49:04.633594 17310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:04.633888 17310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:04.633999 17310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:04.641537 17310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:04.641664 17310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:04.641783 17310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.67
I20260806 02:49:04.651863 17310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.67:44225
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.67
--webserver_port=38321
--tserver_master_addrs=127.1.210.126:45859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.67
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:49:04.653180 17310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:04.654510 17310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:04.668397 17318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:04.668465 17316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:04.668440 17315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:04.669459 17310 server_base.cc:1034] running on GCE node
I20260806 02:49:04.670140 17310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:04.671267 17310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:04.672474 17310 hybrid_clock.cc:648] HybridClock initialized: now 1785984544672406 us; error 59 us; skew 500 ppm
I20260806 02:49:04.672827 17310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:49:04 dist-test-slave-8tdl krb5kdc[15840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984544, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:04.676158 17310 init.cc:377] Logged in from keytab as kudu/127.1.210.67@KRBTEST.COM (short username kudu)
I20260806 02:49:04.677685 17310 webserver.cc:460] Webserver started at http://127.1.210.67:38321/ using document root <none> and password file <none>
I20260806 02:49:04.678408 17310 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:04.678514 17310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:04.684088 17310 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:49:04.687187 17325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:04.688629 17310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:49:04.688804 17310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/wal
uuid: "f24ecb82f42249ee97edfab14084c27c"
format_stamp: "Formatted at 2026-08-06 02:48:58 on dist-test-slave-8tdl"
server_key: "d7cd5a9d507b9dda6592ef516e507b05"
server_key_iv: "4b65943704d813758b31e6d394e0a541"
server_key_version: "encryptionkey@0"
I20260806 02:49:04.689851 17310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:04.716939 17310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:04.721676 17310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:04.721974 17310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:04.723090 17310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:49:04.724589 17310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:49:04.724678 17310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:04.724820 17310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:49:04.724913 17310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:04.760972 17310 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.67:44225
I20260806 02:49:04.761142 17440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.67:44225 every 8 connection(s)
I20260806 02:49:04.762699 17310 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
I20260806 02:49:04.768102 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:04.765963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.67:36085)
0806 02:49:04.766859 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:04.767141 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":317,"threads_started":1}
W20260806 02:49:04.768807 17441 heartbeater.cc:644] Failed to heartbeat to 127.1.210.126:45859 (0 consecutive failures): Network error: Failed to ping master at 127.1.210.126:45859: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
I20260806 02:49:04.769757 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:04.769435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.67:48263)
0806 02:49:04.769587 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:04.769654 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 02:49:04.770735 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:04.770335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.67:46883)
0806 02:49:04.770478 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:04.770577 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260806 02:49:04.771035 17441 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 02:49:04.773295 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 17310
I20260806 02:49:04.798028 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:04.797038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.65:58429)
0806 02:49:04.797617 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:04.797740 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":265,"threads_started":1}
I20260806 02:49:05.284942 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:05.283900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.66:47209)
0806 02:49:05.284618 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:05.284706 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":473,"threads_started":1}
I20260806 02:49:05.772759 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:05.771713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.67:45869)
0806 02:49:05.772400 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:05.772502 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":408,"threads_started":1}
I20260806 02:49:05.800055 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:05.799164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.65:50297)
0806 02:49:05.799740 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:05.799832 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":340,"threads_started":1}
I20260806 02:49:06.287039 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:06.286060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.66:52517)
0806 02:49:06.286623 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:06.286868 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":246,"threads_started":1}
I20260806 02:49:06.774643 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:06.773776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.67:47039)
0806 02:49:06.774361 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:06.774460 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":312,"threads_started":1}
I20260806 02:49:06.802258 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:06.801181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.65:50887)
0806 02:49:06.801834 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:06.801976 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":328,"threads_started":1}
I20260806 02:49:07.289248 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:07.288213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.66:42651)
0806 02:49:07.288853 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:07.288981 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":310,"threads_started":1}
I20260806 02:49:07.776907 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:07.775915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.67:41001)
0806 02:49:07.776581 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:07.776700 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":322,"threads_started":1}
I20260806 02:49:07.804333 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:07.803347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.65:42151)
0806 02:49:07.804038 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:07.804136 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":432,"threads_started":1}
I20260806 02:49:08.291644 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:08.290410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.66:33595)
0806 02:49:08.291249 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:08.291367 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":340,"threads_started":1}
I20260806 02:49:08.778766 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:08.777886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.67:50919)
0806 02:49:08.778411 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:08.778502 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":304,"threads_started":1}
I20260806 02:49:08.806258 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:08.805471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.65:44055)
0806 02:49:08.805965 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:08.806060 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":247,"threads_started":1}
I20260806 02:49:09.293779 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:09.292857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.66:42095)
0806 02:49:09.293430 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:09.293536 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":322,"threads_started":1}
I20260806 02:49:09.780637 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:09.779883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.67:40577)
0806 02:49:09.780363 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:09.780458 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":246,"threads_started":1}
I20260806 02:49:09.808213 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:09.807294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.65:47455)
0806 02:49:09.807871 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:09.807953 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":334,"threads_started":1}
I20260806 02:49:10.295791 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:10.294978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.66:35253)
0806 02:49:10.295516 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:10.295620 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":272,"threads_started":1}
I20260806 02:49:10.782513 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:10.781678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.67:44437)
0806 02:49:10.782240 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:10.782340 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":318,"threads_started":1}
I20260806 02:49:10.810101 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:10.809192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.65:59537)
0806 02:49:10.809774 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:10.809882 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":281,"threads_started":1}
I20260806 02:49:11.297757 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:11.296780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.66:57421)
0806 02:49:11.297418 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:11.297522 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":302,"threads_started":1}
I20260806 02:49:11.784475 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:11.783587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.67:33127)
0806 02:49:11.784151 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:11.784250 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":268,"threads_started":1}
I20260806 02:49:11.812024 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:11.811117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.65:40487)
0806 02:49:11.811711 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:11.811806 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":339,"threads_started":1}
I20260806 02:49:12.299856 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:12.298962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.66:58339)
0806 02:49:12.299498 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:12.299635 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":277,"threads_started":1}
I20260806 02:49:12.786567 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:12.785746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.67:47689)
0806 02:49:12.786298 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:12.786391 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":306,"threads_started":1}
I20260806 02:49:12.813995 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:12.813088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.65:39545)
0806 02:49:12.813648 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:12.813757 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":263,"threads_started":1}
I20260806 02:49:13.301671 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:13.300875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.66:35507)
0806 02:49:13.301372 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:13.301459 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":246,"threads_started":1}
I20260806 02:49:13.788806 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:13.787823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.67:42763)
0806 02:49:13.788545 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:13.788625 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":441,"threads_started":1}
I20260806 02:49:13.815949 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:13.815139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.65:58835)
0806 02:49:13.815666 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:13.815755 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":301,"threads_started":1}
I20260806 02:49:14.303460 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:14.302639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.66:42973)
0806 02:49:14.303198 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:14.303284 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":277,"threads_started":1}
I20260806 02:49:14.790572 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:14.789858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.67:59047)
0806 02:49:14.790316 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:14.790404 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":238,"threads_started":1}
I20260806 02:49:14.817799 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:14.816945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.65:52969)
0806 02:49:14.817496 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:14.817587 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":325,"threads_started":1}
I20260806 02:49:15.305325 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:15.304477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.66:38959)
0806 02:49:15.304979 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:15.305133 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":282,"threads_started":1}
I20260806 02:49:15.792323 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:15.791570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.67:48639)
0806 02:49:15.792042 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:15.792148 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":227,"threads_started":1}
I20260806 02:49:15.819638 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:15.818882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.65:59343)
0806 02:49:15.819380 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:15.819468 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":292,"threads_started":1}
I20260806 02:49:16.307369 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:16.306525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.66:51151)
0806 02:49:16.307092 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:16.307184 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":303,"threads_started":1}
I20260806 02:49:16.794209 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:16.793382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.67:44321)
0806 02:49:16.793857 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:16.794038 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":235,"threads_started":1}
I20260806 02:49:16.821344 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:16.820548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.65:57871)
0806 02:49:16.821040 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:16.821148 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":235,"threads_started":1}
I20260806 02:49:17.309405 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:17.308526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.66:33587)
0806 02:49:17.309065 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:17.309208 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":286,"threads_started":1}
I20260806 02:49:17.796119 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:17.795280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.67:55927)
0806 02:49:17.795785 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:17.795898 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":252,"threads_started":1}
I20260806 02:49:17.823204 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:17.822332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.65:47893)
0806 02:49:17.822886 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:17.822992 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":246,"threads_started":1}
I20260806 02:49:18.311290 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:18.310416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.66:48759)
0806 02:49:18.310994 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:18.311091 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":310,"threads_started":1}
I20260806 02:49:18.797878 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:18.797084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.67:43443)
0806 02:49:18.797587 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:18.797678 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":255,"threads_started":1}
I20260806 02:49:18.825434 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:18.824593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.65:56895)
0806 02:49:18.825141 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:18.825241 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":317,"threads_started":1}
I20260806 02:49:19.313510 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:19.312675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.66:35437)
0806 02:49:19.313217 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:19.313321 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":269,"threads_started":1}
I20260806 02:49:19.786164 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 17008
I20260806 02:49:19.800225 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:19.799322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.67:40399)
0806 02:49:19.799901 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:19.800034 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":320,"threads_started":1}
I20260806 02:49:19.827576 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:19.826645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:45859 (local address 127.1.210.65:51281)
0806 02:49:19.827297 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:19.827405 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:45859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":287,"threads_started":1}
I20260806 02:49:20.046020 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:19.787630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49418 (local address 127.1.210.65:35323)
0806 02:49:19.788337 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:19.788363 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:20.027543 (+239180us) server_negotiation.cc:407] Connection header received
0806 02:49:20.029552 (+ 2009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:20.029562 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:20.030041 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:20.030456 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:20.032226 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:20.033593 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:20.035068 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:20.035967 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:20.036165 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:20.039170 (+ 3005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:20.039179 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:20.039196 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:20.039239 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:20.042297 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:20.043162 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:20.043171 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:20.043181 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:20.043371 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:20.043792 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:20.043797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:20.043801 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:20.044371 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:20.044711 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:20.045203 (+ 492us) server_negotiation.cc:299] Negotiation successful
0806 02:49:20.045698 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":277,"threads_started":1}
I20260806 02:49:20.047710 17168 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:49:20.138074 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 17008
I20260806 02:49:20.161206 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 17176
I20260806 02:49:20.181533 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:20.162240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53980 (local address 127.1.210.66:45613)
0806 02:49:20.162938 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:20.162954 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:20.162996 (+ 42us) server_negotiation.cc:407] Connection header received
0806 02:49:20.164756 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:20.164767 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:20.165249 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:20.165596 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:20.167193 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:20.168568 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:20.170031 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:20.170952 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:20.171114 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:20.174225 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:20.174234 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:20.174251 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:20.174301 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:20.177623 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:20.178379 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:20.178387 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:20.178394 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:20.178579 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:20.179273 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:20.179277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:20.179279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:20.179913 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:20.180194 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:20.180675 (+ 481us) server_negotiation.cc:299] Negotiation successful
0806 02:49:20.181216 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":297,"threads_started":1}
I20260806 02:49:20.183320 17302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:49:20.274224 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 17176
I20260806 02:49:20.296109 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 17310
I20260806 02:49:20.315773 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:20.297304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46288 (local address 127.1.210.67:44225)
0806 02:49:20.297961 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:20.297976 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:20.298018 (+ 42us) server_negotiation.cc:407] Connection header received
0806 02:49:20.299829 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:20.299838 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:20.300353 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:20.300743 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:20.302326 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:20.303738 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:20.305081 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:20.305788 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:20.305996 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:20.308915 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:20.308924 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:20.308941 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:20.308988 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:20.312067 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:20.312858 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:20.312866 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:20.312874 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:20.313062 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:20.313495 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:20.313501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:20.313504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:20.314108 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:20.314437 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:20.314988 (+ 551us) server_negotiation.cc:299] Negotiation successful
0806 02:49:20.315482 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":322,"threads_started":1}
I20260806 02:49:20.317315 17436 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:49:20.403764 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 17310
2026-08-06T02:49:20Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25161 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:49:20 dist-test-slave-8tdl krb5kdc[17499](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:49:20 dist-test-slave-8tdl krb5kdc[17499](info): set up 2 sockets
Aug 06 02:49:20 dist-test-slave-8tdl krb5kdc[17499](info): commencing operation
krb5kdc: starting...
W20260806 02:49:21.546586 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:49:21 dist-test-slave-8tdl krb5kdc[17499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:49:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:49:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
I20260806 02:49:21.856237 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:32935
--webserver_interface=127.1.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:34643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:32935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:49:22.220906 17512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:22.221222 17512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:22.221318 17512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:22.229017 17512 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 02:49:22.229135 17512 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 02:49:22.229188 17512 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 02:49:22.229252 17512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:49:22.229310 17512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:22.229379 17512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:22.229426 17512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:49:22.229521 17512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:49:22.240177 17512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:34643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:32935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:32935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:49:22.241559 17512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:22.242960 17512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:22.256036 17520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:22.256058 17517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:22.255980 17518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:22.256397 17512 server_base.cc:1034] running on GCE node
I20260806 02:49:22.257489 17512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:22.258982 17512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:22.260228 17512 hybrid_clock.cc:648] HybridClock initialized: now 1785984562260154 us; error 37 us; skew 500 ppm
I20260806 02:49:22.260576 17512 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:49:22 dist-test-slave-8tdl krb5kdc[17499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984562, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:22.264051 17512 init.cc:377] Logged in from keytab as kudu/127.1.210.126@KRBTEST.COM (short username kudu)
I20260806 02:49:22.265566 17512 webserver.cc:460] Webserver started at http://127.1.210.126:43525/ using document root <none> and password file <none>
I20260806 02:49:22.266235 17512 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:22.266319 17512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:22.266577 17512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:22.269665 17512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "fab79f09beb9418f9face54c9be3c59c"
format_stamp: "Formatted at 2026-08-06 02:49:22 on dist-test-slave-8tdl"
server_key: "022c7a4947764434fbfbfef604332080"
server_key_iv: "fba9cbe23d2caedad5bacc980a2cc8dd"
server_key_version: "encryptionkey@0"
I20260806 02:49:22.270452 17512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "fab79f09beb9418f9face54c9be3c59c"
format_stamp: "Formatted at 2026-08-06 02:49:22 on dist-test-slave-8tdl"
server_key: "022c7a4947764434fbfbfef604332080"
server_key_iv: "fba9cbe23d2caedad5bacc980a2cc8dd"
server_key_version: "encryptionkey@0"
I20260806 02:49:22.275873 17512 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260806 02:49:22.279893 17527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:22.281509 17512 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:49:22.281730 17512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "fab79f09beb9418f9face54c9be3c59c"
format_stamp: "Formatted at 2026-08-06 02:49:22 on dist-test-slave-8tdl"
server_key: "022c7a4947764434fbfbfef604332080"
server_key_iv: "fba9cbe23d2caedad5bacc980a2cc8dd"
server_key_version: "encryptionkey@0"
I20260806 02:49:22.282002 17512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:22.334839 17512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:22.339942 17512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:22.340263 17512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:22.362089 17512 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:32935
I20260806 02:49:22.362115 17579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:32935 every 8 connection(s)
I20260806 02:49:22.363598 17512 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:49:22.367295 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 17512
I20260806 02:49:22.367559 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:49:22.368470 1865 external_mini_cluster.cc:1428] Setting key 280650636d5c6e1ed1d1d4dc2e190aaa
I20260806 02:49:22.368595 17580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:22.384567 17580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c: Bootstrap starting.
Aug 06 02:49:22 dist-test-slave-8tdl krb5kdc[17499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:49:22.388466 17580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:22.389644 17580 log.cc:824] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:22.392982 17580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c: No bootstrap required, opened a new log
I20260806 02:49:22.395357 17583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:22.370851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53534 (local address 127.1.210.126:32935)
0806 02:49:22.371736 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:22.371748 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:22.371800 (+ 52us) server_negotiation.cc:407] Connection header received
0806 02:49:22.373767 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:22.373790 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:22.374220 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:22.374557 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:22.376640 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:22.378028 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:22.380107 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:22.380866 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:22.381057 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:22.386460 (+ 5403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:22.386469 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:22.386485 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:22.386550 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:22.390222 (+ 3672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:22.391390 (+ 1168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:22.391397 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:22.391405 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:22.391596 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:22.392106 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:22.392111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:22.392113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:22.392685 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:22.392921 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:22.393590 (+ 669us) server_negotiation.cc:299] Negotiation successful
0806 02:49:22.394102 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":400,"threads_started":1}
I20260806 02:49:22.403153 17580 raft_consensus.cc:348] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fab79f09beb9418f9face54c9be3c59c" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 32935 } }
I20260806 02:49:22.403718 17580 raft_consensus.cc:374] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:22.403846 17580 raft_consensus.cc:729] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fab79f09beb9418f9face54c9be3c59c, State: Initialized, Role: FOLLOWER
I20260806 02:49:22.404723 17580 consensus_queue.cc:260] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fab79f09beb9418f9face54c9be3c59c" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 32935 } }
I20260806 02:49:22.405177 17580 raft_consensus.cc:388] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:49:22.405390 17580 raft_consensus.cc:482] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:49:22.405593 17580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:22.409457 17580 raft_consensus.cc:504] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fab79f09beb9418f9face54c9be3c59c" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 32935 } }
I20260806 02:49:22.410288 17580 leader_election.cc:302] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fab79f09beb9418f9face54c9be3c59c; no voters:
I20260806 02:49:22.411001 17580 leader_election.cc:288] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:49:22.411181 17585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:49:22.411886 17585 raft_consensus.cc:686] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [term 1 LEADER]: Becoming Leader. State: Replica: fab79f09beb9418f9face54c9be3c59c, State: Running, Role: LEADER
I20260806 02:49:22.412498 17585 consensus_queue.cc:237] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fab79f09beb9418f9face54c9be3c59c" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 32935 } }
I20260806 02:49:22.413271 17580 sys_catalog.cc:564] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:49:22.416126 17586 sys_catalog.cc:455] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fab79f09beb9418f9face54c9be3c59c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fab79f09beb9418f9face54c9be3c59c" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 32935 } } }
I20260806 02:49:22.416314 17587 sys_catalog.cc:455] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fab79f09beb9418f9face54c9be3c59c. Latest consensus state: current_term: 1 leader_uuid: "fab79f09beb9418f9face54c9be3c59c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fab79f09beb9418f9face54c9be3c59c" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 32935 } } }
I20260806 02:49:22.416651 17586 sys_catalog.cc:458] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:22.416656 17587 sys_catalog.cc:458] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:22.419212 17592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:49:22.424441 17592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:49:22.433305 17592 catalog_manager.cc:1347] Generated new cluster ID: bcfb26b2720d4a18a8ed26ea8248af08
I20260806 02:49:22.433439 17592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:49:22.455601 17592 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:49:22.457577 17592 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:49:22.467509 17592 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c: Generated new TSK 0
I20260806 02:49:22.468632 17592 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab.
Entry for principal kudu/127.1.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab.
I20260806 02:49:22.543527 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.65:0
--local_ip_for_outbound_sockets=127.1.210.65
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:32935
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:34643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:49:22.904572 17606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:22.904929 17606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:22.905092 17606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:22.912731 17606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:22.912932 17606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:22.913136 17606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.65
I20260806 02:49:22.923825 17606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:34643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:32935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.65
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:49:22.925269 17606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:22.926797 17606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:22.941227 17614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:22.941318 17612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:22.941265 17611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:22.942830 17606 server_base.cc:1034] running on GCE node
I20260806 02:49:22.943506 17606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:22.944646 17606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:22.945904 17606 hybrid_clock.cc:648] HybridClock initialized: now 1785984562945826 us; error 61 us; skew 500 ppm
I20260806 02:49:22.946283 17606 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:49:22 dist-test-slave-8tdl krb5kdc[17499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984562, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:22.949877 17606 init.cc:377] Logged in from keytab as kudu/127.1.210.65@KRBTEST.COM (short username kudu)
I20260806 02:49:22.951556 17606 webserver.cc:460] Webserver started at http://127.1.210.65:37947/ using document root <none> and password file <none>
I20260806 02:49:22.952299 17606 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:22.952405 17606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:22.952688 17606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:22.955821 17606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/data/instance:
uuid: "cffa79db50164fc6a5319c74034efccf"
format_stamp: "Formatted at 2026-08-06 02:49:22 on dist-test-slave-8tdl"
server_key: "effd7f0c632917196d71b0d09b756c1e"
server_key_iv: "b263af22d2af1434d8e9bef3b3309654"
server_key_version: "encryptionkey@0"
I20260806 02:49:22.956565 17606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance:
uuid: "cffa79db50164fc6a5319c74034efccf"
format_stamp: "Formatted at 2026-08-06 02:49:22 on dist-test-slave-8tdl"
server_key: "effd7f0c632917196d71b0d09b756c1e"
server_key_iv: "b263af22d2af1434d8e9bef3b3309654"
server_key_version: "encryptionkey@0"
I20260806 02:49:22.962018 17606 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 02:49:22.966131 17621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:22.968039 17606 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:49:22.968257 17606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/wal
uuid: "cffa79db50164fc6a5319c74034efccf"
format_stamp: "Formatted at 2026-08-06 02:49:22 on dist-test-slave-8tdl"
server_key: "effd7f0c632917196d71b0d09b756c1e"
server_key_iv: "b263af22d2af1434d8e9bef3b3309654"
server_key_version: "encryptionkey@0"
I20260806 02:49:22.968494 17606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:22.999748 17606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:23.005529 17606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:23.005900 17606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:23.007155 17606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:49:23.009272 17606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:49:23.009370 17606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:23.009550 17606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:49:23.009630 17606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:23.048543 17606 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.65:41307
I20260806 02:49:23.048601 17735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.65:41307 every 8 connection(s)
I20260806 02:49:23.050004 17606 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
I20260806 02:49:23.056478 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 17606
I20260806 02:49:23.056762 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance
I20260806 02:49:23.057436 1865 external_mini_cluster.cc:1428] Setting key c5d7552649033d33475b9afab15f4634
Aug 06 02:49:23 dist-test-slave-8tdl krb5kdc[17499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984562, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:49:23.074137 17738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:23.053227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:35975 (local address 127.1.210.126:32935)
0806 02:49:23.053737 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:23.053742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:23.055024 (+ 1282us) server_negotiation.cc:407] Connection header received
0806 02:49:23.057132 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:23.057138 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:23.057242 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:23.057369 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:23.059490 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:23.060249 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:23.061927 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:23.062358 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:23.062514 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:23.067062 (+ 4548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:23.067069 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:23.067072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:23.067114 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:23.070099 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:23.071152 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:23.071157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:23.071160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:23.071311 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:23.071907 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:23.071912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:23.071914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:23.072291 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:23.072457 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:23.073527 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0806 02:49:23.073816 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":253,"threads_started":1}
I20260806 02:49:23.075793 17739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:23.053840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.65:35975)
0806 02:49:23.054816 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:23.054887 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:23.056793 (+ 1906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:23.057658 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:23.057668 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:23.058405 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:49:23.059281 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:23.059313 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:23.060448 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:23.060452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:23.061755 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:23.061770 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:23.062635 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:23.062644 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:23.062946 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:23.063775 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:49:23.063805 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:49:23.066759 (+ 2954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:49:23.070299 (+ 3540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:23.070311 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:23.070330 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:23.070987 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:23.071439 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:23.071445 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:23.071449 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:23.071765 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:23.072464 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:49:23.072485 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:49:23.073031 (+ 546us) client_negotiation.cc:769] Sending connection context
0806 02:49:23.073386 (+ 355us) client_negotiation.cc:240] Negotiation successful
0806 02:49:23.074030 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":391,"threads_started":1}
I20260806 02:49:23.077785 17736 heartbeater.cc:347] Connected to a master server at 127.1.210.126:32935
I20260806 02:49:23.078146 17736 heartbeater.cc:464] Registering TS with master...
I20260806 02:49:23.079128 17736 heartbeater.cc:511] Master 127.1.210.126:32935 requested a full tablet report, sending...
I20260806 02:49:23.081929 17544 ts_manager.cc:194] Registered new tserver with Master: cffa79db50164fc6a5319c74034efccf (127.1.210.65:41307)
I20260806 02:49:23.084367 17544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.65@KRBTEST.COM'} at 127.1.210.65:35975
WARNING: no policy specified for kudu/127.1.210.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab.
Entry for principal kudu/127.1.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab.
I20260806 02:49:23.131017 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.66:0
--local_ip_for_outbound_sockets=127.1.210.66
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:32935
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:34643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:49:23.496846 17742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:23.497212 17742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:23.497392 17742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:23.505012 17742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:23.505189 17742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:23.505364 17742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.66
I20260806 02:49:23.515525 17742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:34643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:32935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.66
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:49:23.516912 17742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:23.518323 17742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:23.532420 17750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:23.532400 17748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:23.532711 17747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:23.533833 17742 server_base.cc:1034] running on GCE node
I20260806 02:49:23.534505 17742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:23.535584 17742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:23.536819 17742 hybrid_clock.cc:648] HybridClock initialized: now 1785984563536747 us; error 70 us; skew 500 ppm
I20260806 02:49:23.537156 17742 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:49:23 dist-test-slave-8tdl krb5kdc[17499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984563, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:23.544625 17742 init.cc:377] Logged in from keytab as kudu/127.1.210.66@KRBTEST.COM (short username kudu)
I20260806 02:49:23.546229 17742 webserver.cc:460] Webserver started at http://127.1.210.66:43723/ using document root <none> and password file <none>
I20260806 02:49:23.547042 17742 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:23.547168 17742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:23.547454 17742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:23.550808 17742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/data/instance:
uuid: "eca26f9a66a946a18f4afe233ed0bbec"
format_stamp: "Formatted at 2026-08-06 02:49:23 on dist-test-slave-8tdl"
server_key: "14f7ecf0bce6698a8451bcfe232a07cc"
server_key_iv: "bb85b01eef9fcbf41ea1a15cb98805ec"
server_key_version: "encryptionkey@0"
I20260806 02:49:23.551617 17742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance:
uuid: "eca26f9a66a946a18f4afe233ed0bbec"
format_stamp: "Formatted at 2026-08-06 02:49:23 on dist-test-slave-8tdl"
server_key: "14f7ecf0bce6698a8451bcfe232a07cc"
server_key_iv: "bb85b01eef9fcbf41ea1a15cb98805ec"
server_key_version: "encryptionkey@0"
I20260806 02:49:23.557652 17742 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 02:49:23.561805 17757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:23.563535 17742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:49:23.563742 17742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/wal
uuid: "eca26f9a66a946a18f4afe233ed0bbec"
format_stamp: "Formatted at 2026-08-06 02:49:23 on dist-test-slave-8tdl"
server_key: "14f7ecf0bce6698a8451bcfe232a07cc"
server_key_iv: "bb85b01eef9fcbf41ea1a15cb98805ec"
server_key_version: "encryptionkey@0"
I20260806 02:49:23.563966 17742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:23.589851 17742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:23.595082 17742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:23.595467 17742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:23.596634 17742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:49:23.598115 17742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:49:23.598197 17742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:23.598289 17742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:49:23.598337 17742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:23.634090 17742 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.66:38745
I20260806 02:49:23.634152 17870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.66:38745 every 8 connection(s)
I20260806 02:49:23.635613 17742 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
I20260806 02:49:23.642072 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 17742
I20260806 02:49:23.642268 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance
I20260806 02:49:23.642815 1865 external_mini_cluster.cc:1428] Setting key 3eddc6da96cc43a0ae7b96d409002de6
Aug 06 02:49:23 dist-test-slave-8tdl krb5kdc[17499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984563, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:49:23.660844 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:23.639547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:33507 (local address 127.1.210.126:32935)
0806 02:49:23.640246 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:23.640252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:23.641183 (+ 931us) server_negotiation.cc:407] Connection header received
0806 02:49:23.643363 (+ 2180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:23.643370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:23.643472 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:23.643577 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:23.645390 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:23.646512 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:23.648300 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:23.649075 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:23.649227 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:23.654097 (+ 4870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:23.654103 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:23.654106 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:23.654144 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:23.657050 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:23.657948 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:23.657957 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:23.657964 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:23.658131 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:23.658612 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:23.658620 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:23.658626 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:23.659073 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:23.659243 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:23.660044 (+ 801us) server_negotiation.cc:299] Negotiation successful
0806 02:49:23.660332 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":300,"threads_started":1}
I20260806 02:49:23.661630 17874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:23.640093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.66:33507)
0806 02:49:23.640969 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:23.641049 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:23.643061 (+ 2012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:23.643864 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:23.643874 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:23.644443 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:49:23.645200 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:23.645220 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:23.646743 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:23.646751 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:23.648132 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:23.648149 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:23.649324 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:23.649333 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:23.649658 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:23.650499 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:49:23.650534 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:49:23.653757 (+ 3223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:49:23.657228 (+ 3471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:23.657241 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:23.657260 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:23.657785 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:23.658246 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:23.658250 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:23.658252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:23.658485 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:23.659237 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:49:23.659243 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:49:23.659628 (+ 385us) client_negotiation.cc:769] Sending connection context
0806 02:49:23.659878 (+ 250us) client_negotiation.cc:240] Negotiation successful
0806 02:49:23.660330 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":359,"threads_started":1}
I20260806 02:49:23.663378 17871 heartbeater.cc:347] Connected to a master server at 127.1.210.126:32935
I20260806 02:49:23.663750 17871 heartbeater.cc:464] Registering TS with master...
I20260806 02:49:23.664620 17871 heartbeater.cc:511] Master 127.1.210.126:32935 requested a full tablet report, sending...
I20260806 02:49:23.666373 17544 ts_manager.cc:194] Registered new tserver with Master: eca26f9a66a946a18f4afe233ed0bbec (127.1.210.66:38745)
I20260806 02:49:23.667483 17544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.66@KRBTEST.COM'} at 127.1.210.66:33507
WARNING: no policy specified for kudu/127.1.210.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab.
Entry for principal kudu/127.1.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab.
I20260806 02:49:23.712433 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.67:0
--local_ip_for_outbound_sockets=127.1.210.67
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:32935
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:34643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:49:24.065058 17877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:24.065382 17877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:24.065482 17877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:24.073382 17877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:24.073518 17877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:24.073648 17877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.67
I20260806 02:49:24.083899 17877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:34643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:32935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.67
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:49:24.085269 17877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:24.086711 17877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:49:24.088377 17736 heartbeater.cc:503] Master 127.1.210.126:32935 was elected leader, sending a full tablet report...
W20260806 02:49:24.101931 17885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:24.102082 17883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:24.102105 17882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:24.103237 17877 server_base.cc:1034] running on GCE node
I20260806 02:49:24.103935 17877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:24.105019 17877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:24.106307 17877 hybrid_clock.cc:648] HybridClock initialized: now 1785984564106248 us; error 48 us; skew 500 ppm
I20260806 02:49:24.106772 17877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:49:24 dist-test-slave-8tdl krb5kdc[17499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984564, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:24.110198 17877 init.cc:377] Logged in from keytab as kudu/127.1.210.67@KRBTEST.COM (short username kudu)
I20260806 02:49:24.112154 17877 webserver.cc:460] Webserver started at http://127.1.210.67:37893/ using document root <none> and password file <none>
I20260806 02:49:24.112973 17877 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:24.113152 17877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:24.113507 17877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:24.116793 17877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/data/instance:
uuid: "190f26f416b74325b4abedc2cffb5422"
format_stamp: "Formatted at 2026-08-06 02:49:24 on dist-test-slave-8tdl"
server_key: "787e8043adbd31e3822ddb8c340b7fde"
server_key_iv: "a67518e645cea3b84b1d1da42711a18a"
server_key_version: "encryptionkey@0"
I20260806 02:49:24.117587 17877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance:
uuid: "190f26f416b74325b4abedc2cffb5422"
format_stamp: "Formatted at 2026-08-06 02:49:24 on dist-test-slave-8tdl"
server_key: "787e8043adbd31e3822ddb8c340b7fde"
server_key_iv: "a67518e645cea3b84b1d1da42711a18a"
server_key_version: "encryptionkey@0"
I20260806 02:49:24.123683 17877 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260806 02:49:24.127731 17892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:24.129361 17877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260806 02:49:24.129567 17877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/wal
uuid: "190f26f416b74325b4abedc2cffb5422"
format_stamp: "Formatted at 2026-08-06 02:49:24 on dist-test-slave-8tdl"
server_key: "787e8043adbd31e3822ddb8c340b7fde"
server_key_iv: "a67518e645cea3b84b1d1da42711a18a"
server_key_version: "encryptionkey@0"
I20260806 02:49:24.129771 17877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:24.147409 17877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:24.152194 17877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:24.152522 17877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:24.153678 17877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:49:24.155249 17877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:49:24.155324 17877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:24.155483 17877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:49:24.155571 17877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:24.192149 17877 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.67:38123
I20260806 02:49:24.192251 18006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.67:38123 every 8 connection(s)
I20260806 02:49:24.193562 17877 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
I20260806 02:49:24.195206 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 17877
I20260806 02:49:24.195412 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance
I20260806 02:49:24.195971 1865 external_mini_cluster.cc:1428] Setting key 5254aa6987971bc9a807f1a61e2155f4
Aug 06 02:49:24 dist-test-slave-8tdl krb5kdc[17499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984564, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:49:24.218036 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:24.196721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:35449 (local address 127.1.210.126:32935)
0806 02:49:24.197209 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:24.197214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:24.198496 (+ 1282us) server_negotiation.cc:407] Connection header received
0806 02:49:24.200741 (+ 2245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:24.200746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:24.200821 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:24.200946 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:24.203012 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:24.203909 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:24.205812 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:24.206291 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:24.206403 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:24.210375 (+ 3972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:24.210384 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:24.210389 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:24.210444 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:24.214425 (+ 3981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:24.215406 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:24.215410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:24.215413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:24.215559 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:24.216134 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:24.216138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:24.216140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:24.216470 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:24.216616 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:24.217483 (+ 867us) server_negotiation.cc:299] Negotiation successful
0806 02:49:24.217685 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":234,"threads_started":1}
I20260806 02:49:24.219211 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:24.197399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.67:35449)
0806 02:49:24.198341 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:24.198396 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:24.200433 (+ 2037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:24.201242 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:24.201257 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:24.201821 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:49:24.202737 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:24.202769 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:24.204097 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:24.204101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:24.205636 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:24.205651 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:24.206538 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:24.206544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:24.206834 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:24.207559 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:49:24.207583 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:49:24.210073 (+ 2490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:49:24.214596 (+ 4523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:24.214605 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:24.214623 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:24.215259 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:24.215702 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:24.215706 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:24.215709 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:24.215971 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:24.216626 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:49:24.216633 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:49:24.217059 (+ 426us) client_negotiation.cc:769] Sending connection context
0806 02:49:24.217327 (+ 268us) client_negotiation.cc:240] Negotiation successful
0806 02:49:24.217775 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":328,"threads_started":1}
I20260806 02:49:24.220856 18007 heartbeater.cc:347] Connected to a master server at 127.1.210.126:32935
I20260806 02:49:24.221190 18007 heartbeater.cc:464] Registering TS with master...
I20260806 02:49:24.221916 18007 heartbeater.cc:511] Master 127.1.210.126:32935 requested a full tablet report, sending...
I20260806 02:49:24.223451 17544 ts_manager.cc:194] Registered new tserver with Master: 190f26f416b74325b4abedc2cffb5422 (127.1.210.67:38123)
I20260806 02:49:24.224638 17544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.67@KRBTEST.COM'} at 127.1.210.67:35449
I20260806 02:49:24.226089 1865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:49:24.244261 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:24.230915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53550 (local address 127.1.210.126:32935)
0806 02:49:24.231091 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:24.231097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:24.231528 (+ 431us) server_negotiation.cc:407] Connection header received
0806 02:49:24.231669 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:24.231673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:24.231736 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:24.231823 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:24.233453 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:24.234208 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:24.235710 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:24.236105 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:24.236212 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:24.238788 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:24.238793 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:24.238796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:24.238831 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:24.241622 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:24.242370 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:24.242374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:24.242376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:24.242519 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:24.242959 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:24.242964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:24.242967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:24.243304 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:24.243520 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:24.243843 (+ 323us) server_negotiation.cc:299] Negotiation successful
0806 02:49:24.244027 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:49:24.248792 17544 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:49:24.256258 17544 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:49:24.288084 18021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:24.269790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55140 (local address 127.1.210.66:38745)
0806 02:49:24.270990 (+ 1200us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:24.271009 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:24.271896 (+ 887us) server_negotiation.cc:407] Connection header received
0806 02:49:24.272424 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:24.272434 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:24.272799 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:24.273032 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:24.275926 (+ 2894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:24.278077 (+ 2151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:24.284524 (+ 6447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:24.286301 (+ 1777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:24.286520 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:24.286785 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:24.287549 (+ 764us) server_negotiation.cc:299] Negotiation successful
0806 02:49:24.287745 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1012,"thread_start_us":889,"threads_started":1}
I20260806 02:49:24.288133 18020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:24.269585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:38745 (local address 127.0.0.1:55140)
0806 02:49:24.271696 (+ 2111us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:24.271760 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:24.272110 (+ 350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:24.275042 (+ 2932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:24.275054 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:24.275102 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:24.275646 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:24.275665 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:24.280191 (+ 4526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:24.280201 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:24.284231 (+ 4030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:24.284256 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:24.286551 (+ 2295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:24.286557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:24.286862 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:24.287277 (+ 415us) client_negotiation.cc:769] Sending connection context
0806 02:49:24.287427 (+ 150us) client_negotiation.cc:240] Negotiation successful
0806 02:49:24.287658 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1758,"mutex_wait_us":285,"thread_start_us":1606,"threads_started":1}
I20260806 02:49:24.292080 18025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:24.273157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:38123 (local address 127.0.0.1:45522)
0806 02:49:24.282381 (+ 9224us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:24.282403 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:24.282551 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:24.284216 (+ 1665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:24.284223 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:24.284249 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:24.284625 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:24.284635 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:24.286725 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:24.286730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:24.289076 (+ 2346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:24.289090 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:24.291038 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:24.291044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:24.291242 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:24.291291 (+ 49us) client_negotiation.cc:769] Sending connection context
0806 02:49:24.291485 (+ 194us) client_negotiation.cc:240] Negotiation successful
0806 02:49:24.291627 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9065,"mutex_wait_us":38,"thread_start_us":8890,"threads_started":1}
I20260806 02:49:24.292177 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:24.272666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45522 (local address 127.1.210.67:38123)
0806 02:49:24.283472 (+ 10806us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:24.283482 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:24.283521 (+ 39us) server_negotiation.cc:407] Connection header received
0806 02:49:24.283647 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:24.283657 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:24.283956 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:24.284794 (+ 838us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:24.285002 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:24.286523 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:24.289261 (+ 2738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:24.290810 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:24.290963 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:24.291205 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:24.291578 (+ 373us) server_negotiation.cc:299] Negotiation successful
0806 02:49:24.291749 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10599,"thread_start_us":10422,"threads_started":1}
I20260806 02:49:24.293255 17805 tablet_service.cc:1467] Processing CreateTablet for tablet 0275c3336615481bbb1de1ecc93bafef (DEFAULT_TABLE table=test-table [id=0a8bdfe81352466f8ab685481948d6dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:49:24.294250 18022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:24.270370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:41307 (local address 127.0.0.1:52524)
0806 02:49:24.281569 (+ 11199us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:24.281596 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:24.281759 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:24.286725 (+ 4966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:24.286730 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:24.286760 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:24.287371 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:24.287384 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:24.289619 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:24.289627 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:24.291313 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:24.291327 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:24.293192 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:24.293199 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:24.293421 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:24.293466 (+ 45us) client_negotiation.cc:769] Sending connection context
0806 02:49:24.293564 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 02:49:24.293703 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10980,"mutex_wait_us":54,"thread_start_us":12287,"threads_started":1}
I20260806 02:49:24.295122 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:24.271212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52524 (local address 127.1.210.65:41307)
0806 02:49:24.285982 (+ 14770us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:24.285992 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:24.286024 (+ 32us) server_negotiation.cc:407] Connection header received
0806 02:49:24.286137 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:24.286144 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:24.286398 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:24.286934 (+ 536us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:24.287651 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:24.289446 (+ 1795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:24.291480 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:24.292918 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:24.293107 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:24.293330 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:24.294531 (+ 1201us) server_negotiation.cc:299] Negotiation successful
0806 02:49:24.294753 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14589,"thread_start_us":14415,"threads_started":1}
I20260806 02:49:24.296082 17805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0275c3336615481bbb1de1ecc93bafef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:24.296334 17941 tablet_service.cc:1467] Processing CreateTablet for tablet 0275c3336615481bbb1de1ecc93bafef (DEFAULT_TABLE table=test-table [id=0a8bdfe81352466f8ab685481948d6dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:49:24.297583 17941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0275c3336615481bbb1de1ecc93bafef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:24.298959 17669 tablet_service.cc:1467] Processing CreateTablet for tablet 0275c3336615481bbb1de1ecc93bafef (DEFAULT_TABLE table=test-table [id=0a8bdfe81352466f8ab685481948d6dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:49:24.300470 17669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0275c3336615481bbb1de1ecc93bafef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:24.309087 18026 tablet_bootstrap.cc:492] T 0275c3336615481bbb1de1ecc93bafef P 190f26f416b74325b4abedc2cffb5422: Bootstrap starting.
I20260806 02:49:24.312683 18026 tablet_bootstrap.cc:654] T 0275c3336615481bbb1de1ecc93bafef P 190f26f416b74325b4abedc2cffb5422: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:24.313890 18026 log.cc:824] T 0275c3336615481bbb1de1ecc93bafef P 190f26f416b74325b4abedc2cffb5422: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:24.315670 18027 tablet_bootstrap.cc:492] T 0275c3336615481bbb1de1ecc93bafef P eca26f9a66a946a18f4afe233ed0bbec: Bootstrap starting.
I20260806 02:49:24.317338 18029 tablet_bootstrap.cc:492] T 0275c3336615481bbb1de1ecc93bafef P cffa79db50164fc6a5319c74034efccf: Bootstrap starting.
I20260806 02:49:24.317693 18026 tablet_bootstrap.cc:492] T 0275c3336615481bbb1de1ecc93bafef P 190f26f416b74325b4abedc2cffb5422: No bootstrap required, opened a new log
I20260806 02:49:24.318043 18026 ts_tablet_manager.cc:1397] T 0275c3336615481bbb1de1ecc93bafef P 190f26f416b74325b4abedc2cffb5422: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260806 02:49:24.319566 18027 tablet_bootstrap.cc:654] T 0275c3336615481bbb1de1ecc93bafef P eca26f9a66a946a18f4afe233ed0bbec: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:24.320634 18029 tablet_bootstrap.cc:654] T 0275c3336615481bbb1de1ecc93bafef P cffa79db50164fc6a5319c74034efccf: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:24.320868 18027 log.cc:824] T 0275c3336615481bbb1de1ecc93bafef P eca26f9a66a946a18f4afe233ed0bbec: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:24.321808 18029 log.cc:824] T 0275c3336615481bbb1de1ecc93bafef P cffa79db50164fc6a5319c74034efccf: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:24.324853 18027 tablet_bootstrap.cc:492] T 0275c3336615481bbb1de1ecc93bafef P eca26f9a66a946a18f4afe233ed0bbec: No bootstrap required, opened a new log
I20260806 02:49:24.325330 18027 ts_tablet_manager.cc:1397] T 0275c3336615481bbb1de1ecc93bafef P eca26f9a66a946a18f4afe233ed0bbec: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260806 02:49:24.329407 18029 tablet_bootstrap.cc:492] T 0275c3336615481bbb1de1ecc93bafef P cffa79db50164fc6a5319c74034efccf: No bootstrap required, opened a new log
I20260806 02:49:24.329712 18029 ts_tablet_manager.cc:1397] T 0275c3336615481bbb1de1ecc93bafef P cffa79db50164fc6a5319c74034efccf: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260806 02:49:24.329458 18026 raft_consensus.cc:348] T 0275c3336615481bbb1de1ecc93bafef P 190f26f416b74325b4abedc2cffb5422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca26f9a66a946a18f4afe233ed0bbec" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 38745 } } peers { permanent_uuid: "190f26f416b74325b4abedc2cffb5422" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38123 } } peers { permanent_uuid: "cffa79db50164fc6a5319c74034efccf" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 41307 } }
I20260806 02:49:24.329962 18026 raft_consensus.cc:374] T 0275c3336615481bbb1de1ecc93bafef P 190f26f416b74325b4abedc2cffb5422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:24.330073 18026 raft_consensus.cc:729] T 0275c3336615481bbb1de1ecc93bafef P 190f26f416b74325b4abedc2cffb5422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 190f26f416b74325b4abedc2cffb5422, State: Initialized, Role: FOLLOWER
I20260806 02:49:24.330766 18026 consensus_queue.cc:260] T 0275c3336615481bbb1de1ecc93bafef P 190f26f416b74325b4abedc2cffb5422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca26f9a66a946a18f4afe233ed0bbec" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 38745 } } peers { permanent_uuid: "190f26f416b74325b4abedc2cffb5422" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38123 } } peers { permanent_uuid: "cffa79db50164fc6a5319c74034efccf" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 41307 } }
I20260806 02:49:24.334265 18007 heartbeater.cc:503] Master 127.1.210.126:32935 was elected leader, sending a full tablet report...
I20260806 02:49:24.335160 18026 ts_tablet_manager.cc:1428] T 0275c3336615481bbb1de1ecc93bafef P 190f26f416b74325b4abedc2cffb5422: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260806 02:49:24.337240 18027 raft_consensus.cc:348] T 0275c3336615481bbb1de1ecc93bafef P eca26f9a66a946a18f4afe233ed0bbec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca26f9a66a946a18f4afe233ed0bbec" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 38745 } } peers { permanent_uuid: "190f26f416b74325b4abedc2cffb5422" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38123 } } peers { permanent_uuid: "cffa79db50164fc6a5319c74034efccf" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 41307 } }
I20260806 02:49:24.337766 18027 raft_consensus.cc:374] T 0275c3336615481bbb1de1ecc93bafef P eca26f9a66a946a18f4afe233ed0bbec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:24.337862 18027 raft_consensus.cc:729] T 0275c3336615481bbb1de1ecc93bafef P eca26f9a66a946a18f4afe233ed0bbec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eca26f9a66a946a18f4afe233ed0bbec, State: Initialized, Role: FOLLOWER
I20260806 02:49:24.338480 18027 consensus_queue.cc:260] T 0275c3336615481bbb1de1ecc93bafef P eca26f9a66a946a18f4afe233ed0bbec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca26f9a66a946a18f4afe233ed0bbec" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 38745 } } peers { permanent_uuid: "190f26f416b74325b4abedc2cffb5422" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38123 } } peers { permanent_uuid: "cffa79db50164fc6a5319c74034efccf" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 41307 } }
I20260806 02:49:24.339886 17871 heartbeater.cc:503] Master 127.1.210.126:32935 was elected leader, sending a full tablet report...
I20260806 02:49:24.340467 18027 ts_tablet_manager.cc:1428] T 0275c3336615481bbb1de1ecc93bafef P eca26f9a66a946a18f4afe233ed0bbec: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260806 02:49:24.341682 18029 raft_consensus.cc:348] T 0275c3336615481bbb1de1ecc93bafef P cffa79db50164fc6a5319c74034efccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca26f9a66a946a18f4afe233ed0bbec" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 38745 } } peers { permanent_uuid: "190f26f416b74325b4abedc2cffb5422" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38123 } } peers { permanent_uuid: "cffa79db50164fc6a5319c74034efccf" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 41307 } }
I20260806 02:49:24.342039 18029 raft_consensus.cc:374] T 0275c3336615481bbb1de1ecc93bafef P cffa79db50164fc6a5319c74034efccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:24.342167 18029 raft_consensus.cc:729] T 0275c3336615481bbb1de1ecc93bafef P cffa79db50164fc6a5319c74034efccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cffa79db50164fc6a5319c74034efccf, State: Initialized, Role: FOLLOWER
I20260806 02:49:24.342825 18029 consensus_queue.cc:260] T 0275c3336615481bbb1de1ecc93bafef P cffa79db50164fc6a5319c74034efccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca26f9a66a946a18f4afe233ed0bbec" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 38745 } } peers { permanent_uuid: "190f26f416b74325b4abedc2cffb5422" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38123 } } peers { permanent_uuid: "cffa79db50164fc6a5319c74034efccf" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 41307 } }
I20260806 02:49:24.344622 18029 ts_tablet_manager.cc:1428] T 0275c3336615481bbb1de1ecc93bafef P cffa79db50164fc6a5319c74034efccf: Time spent starting tablet: real 0.015s user 0.012s sys 0.001s
W20260806 02:49:24.388418 17872 tablet.cc:2369] T 0275c3336615481bbb1de1ecc93bafef P eca26f9a66a946a18f4afe233ed0bbec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:49:24.445410 18008 tablet.cc:2369] T 0275c3336615481bbb1de1ecc93bafef P 190f26f416b74325b4abedc2cffb5422: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:49:24.553251 17737 tablet.cc:2369] T 0275c3336615481bbb1de1ecc93bafef P cffa79db50164fc6a5319c74034efccf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:49:24.650197 18034 raft_consensus.cc:482] T 0275c3336615481bbb1de1ecc93bafef P cffa79db50164fc6a5319c74034efccf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:49:24.650554 18034 raft_consensus.cc:504] T 0275c3336615481bbb1de1ecc93bafef P cffa79db50164fc6a5319c74034efccf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca26f9a66a946a18f4afe233ed0bbec" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 38745 } } peers { permanent_uuid: "190f26f416b74325b4abedc2cffb5422" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38123 } } peers { permanent_uuid: "cffa79db50164fc6a5319c74034efccf" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 41307 } }
I20260806 02:49:24.652405 18034 leader_election.cc:288] T 0275c3336615481bbb1de1ecc93bafef P cffa79db50164fc6a5319c74034efccf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eca26f9a66a946a18f4afe233ed0bbec (127.1.210.66:38745), 190f26f416b74325b4abedc2cffb5422 (127.1.210.67:38123)
I20260806 02:49:24.659880 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:24.653649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:36537 (local address 127.1.210.67:38123)
0806 02:49:24.653891 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:24.653900 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:24.654257 (+ 357us) server_negotiation.cc:407] Connection header received
0806 02:49:24.654437 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:24.654441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:24.654496 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:24.654566 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:24.655237 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:24.656032 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:24.657916 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:24.658837 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:24.659032 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:24.659116 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:24.659470 (+ 354us) server_negotiation.cc:299] Negotiation successful
0806 02:49:24.659598 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 02:49:24.660116 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:24.653537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:38123 (local address 127.1.210.65:36537)
0806 02:49:24.654124 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:24.654139 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:24.654282 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:24.654602 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:24.654610 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:24.654641 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:24.655097 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:24.655105 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:24.656177 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:24.656183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:24.657745 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:24.657760 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:24.659097 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:24.659103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:24.659275 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:24.659315 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 02:49:24.659423 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 02:49:24.659568 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":6,"thread_start_us":301,"threads_started":1}
I20260806 02:49:24.660137 18035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:24.652759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:38745 (local address 127.1.210.65:55055)
0806 02:49:24.653421 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:24.653439 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:24.653624 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:24.654238 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:24.654243 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:24.654331 (+ 88us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:24.654834 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:24.654851 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:24.656047 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:24.656054 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:24.657748 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:24.657763 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:24.659138 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:24.659146 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:24.659334 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:24.659357 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 02:49:24.659468 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 02:49:24.659604 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":58,"spinlock_wait_cycles":24448,"thread_start_us":311,"threads_started":1}
I20260806 02:49:24.659924 18021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:24.653126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:55055 (local address 127.1.210.66:38745)
0806 02:49:24.653361 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:24.653367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:24.653562 (+ 195us) server_negotiation.cc:407] Connection header received
0806 02:49:24.654012 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:24.654019 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:24.654099 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:24.654182 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:24.655016 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:24.655913 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:24.657944 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:24.658836 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:24.659017 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:24.659118 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:24.659497 (+ 379us) server_negotiation.cc:299] Negotiation successful
0806 02:49:24.659629 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 02:49:24.661193 17825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0275c3336615481bbb1de1ecc93bafef" candidate_uuid: "cffa79db50164fc6a5319c74034efccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eca26f9a66a946a18f4afe233ed0bbec" is_pre_election: true
I20260806 02:49:24.661254 17961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0275c3336615481bbb1de1ecc93bafef" candidate_uuid: "cffa79db50164fc6a5319c74034efccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "190f26f416b74325b4abedc2cffb5422" is_pre_election: true
I20260806 02:49:24.661707 17961 raft_consensus.cc:2454] T 0275c3336615481bbb1de1ecc93bafef P 190f26f416b74325b4abedc2cffb5422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cffa79db50164fc6a5319c74034efccf in term 0.
I20260806 02:49:24.661706 17825 raft_consensus.cc:2454] T 0275c3336615481bbb1de1ecc93bafef P eca26f9a66a946a18f4afe233ed0bbec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cffa79db50164fc6a5319c74034efccf in term 0.
I20260806 02:49:24.662362 17625 leader_election.cc:302] T 0275c3336615481bbb1de1ecc93bafef P cffa79db50164fc6a5319c74034efccf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 190f26f416b74325b4abedc2cffb5422, cffa79db50164fc6a5319c74034efccf; no voters:
I20260806 02:49:24.662899 18034 raft_consensus.cc:2781] T 0275c3336615481bbb1de1ecc93bafef P cffa79db50164fc6a5319c74034efccf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:49:24.663089 18034 raft_consensus.cc:482] T 0275c3336615481bbb1de1ecc93bafef P cffa79db50164fc6a5319c74034efccf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:49:24.663215 18034 raft_consensus.cc:3037] T 0275c3336615481bbb1de1ecc93bafef P cffa79db50164fc6a5319c74034efccf [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:24.665620 18034 raft_consensus.cc:504] T 0275c3336615481bbb1de1ecc93bafef P cffa79db50164fc6a5319c74034efccf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca26f9a66a946a18f4afe233ed0bbec" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 38745 } } peers { permanent_uuid: "190f26f416b74325b4abedc2cffb5422" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38123 } } peers { permanent_uuid: "cffa79db50164fc6a5319c74034efccf" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 41307 } }
I20260806 02:49:24.666286 18034 leader_election.cc:288] T 0275c3336615481bbb1de1ecc93bafef P cffa79db50164fc6a5319c74034efccf [CANDIDATE]: Term 1 election: Requested vote from peers eca26f9a66a946a18f4afe233ed0bbec (127.1.210.66:38745), 190f26f416b74325b4abedc2cffb5422 (127.1.210.67:38123)
I20260806 02:49:24.667078 17825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0275c3336615481bbb1de1ecc93bafef" candidate_uuid: "cffa79db50164fc6a5319c74034efccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eca26f9a66a946a18f4afe233ed0bbec"
I20260806 02:49:24.667189 17961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0275c3336615481bbb1de1ecc93bafef" candidate_uuid: "cffa79db50164fc6a5319c74034efccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "190f26f416b74325b4abedc2cffb5422"
I20260806 02:49:24.667309 17825 raft_consensus.cc:3037] T 0275c3336615481bbb1de1ecc93bafef P eca26f9a66a946a18f4afe233ed0bbec [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:24.667451 17961 raft_consensus.cc:3037] T 0275c3336615481bbb1de1ecc93bafef P 190f26f416b74325b4abedc2cffb5422 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:24.669854 17825 raft_consensus.cc:2454] T 0275c3336615481bbb1de1ecc93bafef P eca26f9a66a946a18f4afe233ed0bbec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cffa79db50164fc6a5319c74034efccf in term 1.
I20260806 02:49:24.670257 17961 raft_consensus.cc:2454] T 0275c3336615481bbb1de1ecc93bafef P 190f26f416b74325b4abedc2cffb5422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cffa79db50164fc6a5319c74034efccf in term 1.
I20260806 02:49:24.670451 17625 leader_election.cc:302] T 0275c3336615481bbb1de1ecc93bafef P cffa79db50164fc6a5319c74034efccf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cffa79db50164fc6a5319c74034efccf, eca26f9a66a946a18f4afe233ed0bbec; no voters:
I20260806 02:49:24.670840 18034 raft_consensus.cc:2781] T 0275c3336615481bbb1de1ecc93bafef P cffa79db50164fc6a5319c74034efccf [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:49:24.671355 18034 raft_consensus.cc:686] T 0275c3336615481bbb1de1ecc93bafef P cffa79db50164fc6a5319c74034efccf [term 1 LEADER]: Becoming Leader. State: Replica: cffa79db50164fc6a5319c74034efccf, State: Running, Role: LEADER
I20260806 02:49:24.671870 18034 consensus_queue.cc:237] T 0275c3336615481bbb1de1ecc93bafef P cffa79db50164fc6a5319c74034efccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca26f9a66a946a18f4afe233ed0bbec" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 38745 } } peers { permanent_uuid: "190f26f416b74325b4abedc2cffb5422" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38123 } } peers { permanent_uuid: "cffa79db50164fc6a5319c74034efccf" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 41307 } }
I20260806 02:49:24.677194 17544 catalog_manager.cc:5495] T 0275c3336615481bbb1de1ecc93bafef P cffa79db50164fc6a5319c74034efccf reported cstate change: term changed from 0 to 1, leader changed from <none> to cffa79db50164fc6a5319c74034efccf (127.1.210.65). New cstate: current_term: 1 leader_uuid: "cffa79db50164fc6a5319c74034efccf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eca26f9a66a946a18f4afe233ed0bbec" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 38745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "190f26f416b74325b4abedc2cffb5422" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cffa79db50164fc6a5319c74034efccf" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 41307 } health_report { overall_health: HEALTHY } } }
I20260806 02:49:24.742859 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:24.736677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52530 (local address 127.1.210.65:41307)
0806 02:49:24.736877 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:24.736882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:24.736900 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:49:24.737169 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:24.737173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:24.737255 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:24.737385 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:49:24.737809 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:24.738594 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:24.740263 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:24.740704 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:24.740808 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:24.741456 (+ 648us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:49:24.742072 (+ 616us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:49:24.742207 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:24.742377 (+ 170us) server_negotiation.cc:299] Negotiation successful
0806 02:49:24.742464 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 02:49:24.752707 17961 raft_consensus.cc:1261] T 0275c3336615481bbb1de1ecc93bafef P 190f26f416b74325b4abedc2cffb5422 [term 1 FOLLOWER]: Refusing update from remote peer cffa79db50164fc6a5319c74034efccf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:49:24.752866 17825 raft_consensus.cc:1261] T 0275c3336615481bbb1de1ecc93bafef P eca26f9a66a946a18f4afe233ed0bbec [term 1 FOLLOWER]: Refusing update from remote peer cffa79db50164fc6a5319c74034efccf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:49:24.754005 18037 consensus_queue.cc:1035] T 0275c3336615481bbb1de1ecc93bafef P cffa79db50164fc6a5319c74034efccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "190f26f416b74325b4abedc2cffb5422" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:49:24.754452 18034 consensus_queue.cc:1035] T 0275c3336615481bbb1de1ecc93bafef P cffa79db50164fc6a5319c74034efccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "eca26f9a66a946a18f4afe233ed0bbec" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:49:24.770593 18041 mvcc.cc:205] Tried to move back new op lower bound from 7315392777215893504 to 7315392776899907584. Current Snapshot: MvccSnapshot[applied={T|T < 7315392777215893504}]
I20260806 02:49:24.785218 17544 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:49:24.785526 17544 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:49:24.789719 17544 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c: Sending DeleteTablet for 3 replicas of tablet 0275c3336615481bbb1de1ecc93bafef
I20260806 02:49:24.791250 17805 tablet_service.cc:1514] Processing DeleteTablet for tablet 0275c3336615481bbb1de1ecc93bafef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:49:24 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:55140
I20260806 02:49:24.791615 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 17606
I20260806 02:49:24.791527 17941 tablet_service.cc:1514] Processing DeleteTablet for tablet 0275c3336615481bbb1de1ecc93bafef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:49:24 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:45522
I20260806 02:49:24.792024 17669 tablet_service.cc:1514] Processing DeleteTablet for tablet 0275c3336615481bbb1de1ecc93bafef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:49:24 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:52524
I20260806 02:49:24.803303 18052 tablet_replica.cc:330] T 0275c3336615481bbb1de1ecc93bafef P cffa79db50164fc6a5319c74034efccf: stopping tablet replica
I20260806 02:49:24.803787 18050 tablet_replica.cc:330] T 0275c3336615481bbb1de1ecc93bafef P eca26f9a66a946a18f4afe233ed0bbec: stopping tablet replica
I20260806 02:49:24.803795 18052 raft_consensus.cc:2229] T 0275c3336615481bbb1de1ecc93bafef P cffa79db50164fc6a5319c74034efccf [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:49:24.804286 18050 raft_consensus.cc:2229] T 0275c3336615481bbb1de1ecc93bafef P eca26f9a66a946a18f4afe233ed0bbec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:49:24.804565 18051 tablet_replica.cc:330] T 0275c3336615481bbb1de1ecc93bafef P 190f26f416b74325b4abedc2cffb5422: stopping tablet replica
I20260806 02:49:24.804700 18052 raft_consensus.cc:2258] T 0275c3336615481bbb1de1ecc93bafef P cffa79db50164fc6a5319c74034efccf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:49:24.804843 18050 raft_consensus.cc:2258] T 0275c3336615481bbb1de1ecc93bafef P eca26f9a66a946a18f4afe233ed0bbec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:49:24.805004 18051 raft_consensus.cc:2229] T 0275c3336615481bbb1de1ecc93bafef P 190f26f416b74325b4abedc2cffb5422 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:49:24.805460 18051 raft_consensus.cc:2258] T 0275c3336615481bbb1de1ecc93bafef P 190f26f416b74325b4abedc2cffb5422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:49:24.807633 18052 ts_tablet_manager.cc:1905] T 0275c3336615481bbb1de1ecc93bafef P cffa79db50164fc6a5319c74034efccf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:49:24.807626 18051 ts_tablet_manager.cc:1905] T 0275c3336615481bbb1de1ecc93bafef P 190f26f416b74325b4abedc2cffb5422: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:49:24.808099 18050 ts_tablet_manager.cc:1905] T 0275c3336615481bbb1de1ecc93bafef P eca26f9a66a946a18f4afe233ed0bbec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:49:24.815097 18051 ts_tablet_manager.cc:1918] T 0275c3336615481bbb1de1ecc93bafef P 190f26f416b74325b4abedc2cffb5422: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:49:24.815280 18051 log.cc:1196] T 0275c3336615481bbb1de1ecc93bafef P 190f26f416b74325b4abedc2cffb5422: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/wal/wals/0275c3336615481bbb1de1ecc93bafef
I20260806 02:49:24.815840 18051 ts_tablet_manager.cc:1939] T 0275c3336615481bbb1de1ecc93bafef P 190f26f416b74325b4abedc2cffb5422: Deleting consensus metadata
Aug 06 02:49:24 dist-test-slave-8tdl krb5kdc[17499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.65@KRBTEST.COM
I20260806 02:49:24.817093 18050 ts_tablet_manager.cc:1918] T 0275c3336615481bbb1de1ecc93bafef P eca26f9a66a946a18f4afe233ed0bbec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:49:24.817276 18050 log.cc:1196] T 0275c3336615481bbb1de1ecc93bafef P eca26f9a66a946a18f4afe233ed0bbec: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/wal/wals/0275c3336615481bbb1de1ecc93bafef
I20260806 02:49:24.817848 18050 ts_tablet_manager.cc:1939] T 0275c3336615481bbb1de1ecc93bafef P eca26f9a66a946a18f4afe233ed0bbec: Deleting consensus metadata
I20260806 02:49:24.817996 17531 catalog_manager.cc:4841] TS 190f26f416b74325b4abedc2cffb5422 (127.1.210.67:38123): tablet 0275c3336615481bbb1de1ecc93bafef (table test-table [id=0a8bdfe81352466f8ab685481948d6dd]) successfully deleted
I20260806 02:49:24.819624 17531 catalog_manager.cc:4841] TS eca26f9a66a946a18f4afe233ed0bbec (127.1.210.66:38745): tablet 0275c3336615481bbb1de1ecc93bafef (table test-table [id=0a8bdfe81352466f8ab685481948d6dd]) successfully deleted
I20260806 02:49:24.820020 18052 ts_tablet_manager.cc:1918] T 0275c3336615481bbb1de1ecc93bafef P cffa79db50164fc6a5319c74034efccf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:49:24.820257 18052 log.cc:1196] T 0275c3336615481bbb1de1ecc93bafef P cffa79db50164fc6a5319c74034efccf: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/wal/wals/0275c3336615481bbb1de1ecc93bafef
I20260806 02:49:24.820775 18052 ts_tablet_manager.cc:1939] T 0275c3336615481bbb1de1ecc93bafef P cffa79db50164fc6a5319c74034efccf: Deleting consensus metadata
I20260806 02:49:24.822466 17530 catalog_manager.cc:4841] TS cffa79db50164fc6a5319c74034efccf (127.1.210.65:41307): tablet 0275c3336615481bbb1de1ecc93bafef (table test-table [id=0a8bdfe81352466f8ab685481948d6dd]) successfully deleted
I20260806 02:49:24.826763 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:24.793107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52540 (local address 127.1.210.65:41307)
0806 02:49:24.793352 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:24.793361 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:24.803505 (+ 10144us) server_negotiation.cc:407] Connection header received
0806 02:49:24.804197 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:24.804201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:24.804300 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:24.804417 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:24.807177 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:24.808316 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:24.810524 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:24.811210 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:24.811308 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:24.818509 (+ 7201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:24.818528 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:24.818547 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:24.818609 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:24.823442 (+ 4833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:24.824272 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:24.824283 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:24.824286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:24.824470 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:24.824907 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:24.824917 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:24.824921 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:24.825407 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:24.825733 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:24.826089 (+ 356us) server_negotiation.cc:299] Negotiation successful
0806 02:49:24.826364 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 02:49:24.827556 17731 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:49:24.929836 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 17606
I20260806 02:49:24.952539 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 17742
Aug 06 02:49:24 dist-test-slave-8tdl krb5kdc[17499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.66@KRBTEST.COM
I20260806 02:49:24.969527 18021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:24.953720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55142 (local address 127.1.210.66:38745)
0806 02:49:24.953960 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:24.953964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:24.953980 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:49:24.954040 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:24.954044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:24.954104 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:24.954234 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:24.955733 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:24.956427 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:24.957879 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:24.958291 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:24.958432 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:24.963209 (+ 4777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:24.963225 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:24.963242 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:24.963281 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:24.966266 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:24.967191 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:24.967200 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:24.967202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:24.967344 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:24.967788 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:24.967793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:24.967796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:24.968307 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:24.968572 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:24.968935 (+ 363us) server_negotiation.cc:299] Negotiation successful
0806 02:49:24.969182 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 02:49:24.970415 17866 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:49:25.063545 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 17742
I20260806 02:49:25.082520 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 17877
Aug 06 02:49:25 dist-test-slave-8tdl krb5kdc[17499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.67@KRBTEST.COM
I20260806 02:49:25.100400 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:25.083584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45536 (local address 127.1.210.67:38123)
0806 02:49:25.083828 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:25.083835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:25.083859 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:49:25.084062 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:25.084066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:25.084142 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:25.084282 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:25.085859 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:25.086980 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:25.088398 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:25.088821 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:25.088940 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:25.093835 (+ 4895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:25.093851 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:25.093868 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:25.093960 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:25.097339 (+ 3379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:25.098157 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:25.098167 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:25.098171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:25.098348 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:25.098987 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:25.098991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:25.098994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:25.099377 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:25.099621 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:25.099933 (+ 312us) server_negotiation.cc:299] Negotiation successful
0806 02:49:25.100117 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260806 02:49:25.101200 18002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:49:25.192548 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 17877
I20260806 02:49:25.211215 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 17512
I20260806 02:49:25.212702 17575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:49:25.332010 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 17512
I20260806 02:49:45.359423 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:32935
--webserver_interface=127.1.210.126
--webserver_port=43525
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:34643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:32935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:49:45.730180 18058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:45.730559 18058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:45.730754 18058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:45.739082 18058 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 02:49:45.739249 18058 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 02:49:45.739352 18058 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 02:49:45.739430 18058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:49:45.739534 18058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:45.739645 18058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:45.739737 18058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:49:45.739818 18058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:49:45.751307 18058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:34643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:32935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:32935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=43525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:49:45.752769 18058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:45.754218 18058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:45.782833 18064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:45.782874 18067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:45.782905 18065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:45.783753 18058 server_base.cc:1034] running on GCE node
I20260806 02:49:45.784833 18058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:45.786615 18058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:45.787940 18058 hybrid_clock.cc:648] HybridClock initialized: now 1785984585787866 us; error 42 us; skew 500 ppm
I20260806 02:49:45.788408 18058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:49:45 dist-test-slave-8tdl krb5kdc[17499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984585, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:45.797850 18058 init.cc:377] Logged in from keytab as kudu/127.1.210.126@KRBTEST.COM (short username kudu)
I20260806 02:49:45.800030 18058 webserver.cc:460] Webserver started at http://127.1.210.126:43525/ using document root <none> and password file <none>
I20260806 02:49:45.801051 18058 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:45.801252 18058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:45.808319 18058 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:49:45.811883 18074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:45.813510 18058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:49:45.813740 18058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "fab79f09beb9418f9face54c9be3c59c"
format_stamp: "Formatted at 2026-08-06 02:49:22 on dist-test-slave-8tdl"
server_key: "022c7a4947764434fbfbfef604332080"
server_key_iv: "fba9cbe23d2caedad5bacc980a2cc8dd"
server_key_version: "encryptionkey@0"
I20260806 02:49:45.815027 18058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:45.835841 18058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:45.840969 18058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:45.841411 18058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:45.863024 18058 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:32935
I20260806 02:49:45.863059 18127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:32935 every 8 connection(s)
I20260806 02:49:45.864751 18058 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:49:45.872522 18128 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:49:45.872941 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 18058
I20260806 02:49:45.873823 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.65:41307
--local_ip_for_outbound_sockets=127.1.210.65
--tserver_master_addrs=127.1.210.126:32935
--webserver_port=37947
--webserver_interface=127.1.210.65
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:34643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:49:45.876380 18128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c: Bootstrap starting.
I20260806 02:49:45.898368 18128 log.cc:824] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:45.912097 18128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:49:45.912761 18128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c: Bootstrap complete.
I20260806 02:49:45.923156 18128 raft_consensus.cc:348] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fab79f09beb9418f9face54c9be3c59c" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 32935 } }
I20260806 02:49:45.924197 18128 raft_consensus.cc:729] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fab79f09beb9418f9face54c9be3c59c, State: Initialized, Role: FOLLOWER
I20260806 02:49:45.924896 18128 consensus_queue.cc:260] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fab79f09beb9418f9face54c9be3c59c" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 32935 } }
I20260806 02:49:45.925220 18128 raft_consensus.cc:388] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:49:45.925375 18128 raft_consensus.cc:482] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:49:45.925580 18128 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:49:45.930161 18128 raft_consensus.cc:504] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fab79f09beb9418f9face54c9be3c59c" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 32935 } }
I20260806 02:49:45.930836 18128 leader_election.cc:302] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fab79f09beb9418f9face54c9be3c59c; no voters:
I20260806 02:49:45.931681 18128 leader_election.cc:288] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 02:49:45.931834 18133 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [term 2 FOLLOWER]: Leader election won for term 2
I20260806 02:49:45.933274 18133 raft_consensus.cc:686] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [term 2 LEADER]: Becoming Leader. State: Replica: fab79f09beb9418f9face54c9be3c59c, State: Running, Role: LEADER
I20260806 02:49:45.934011 18133 consensus_queue.cc:237] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fab79f09beb9418f9face54c9be3c59c" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 32935 } }
I20260806 02:49:45.934393 18128 sys_catalog.cc:564] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:49:45.937038 18135 sys_catalog.cc:455] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fab79f09beb9418f9face54c9be3c59c. Latest consensus state: current_term: 2 leader_uuid: "fab79f09beb9418f9face54c9be3c59c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fab79f09beb9418f9face54c9be3c59c" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 32935 } } }
I20260806 02:49:45.937705 18135 sys_catalog.cc:458] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:45.937543 18134 sys_catalog.cc:455] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fab79f09beb9418f9face54c9be3c59c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fab79f09beb9418f9face54c9be3c59c" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 32935 } } }
I20260806 02:49:45.942736 18134 sys_catalog.cc:458] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:45.953703 18139 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:49:45.964219 18139 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:49:45.966015 18139 catalog_manager.cc:1259] Loaded cluster ID: bcfb26b2720d4a18a8ed26ea8248af08
I20260806 02:49:45.966224 18139 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:49:45.971184 18139 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:49:45.973203 18139 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c: Loaded TSK: 0
I20260806 02:49:46.011188 18139 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c: Generated new TSK 1
I20260806 02:49:46.011684 18139 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 02:49:46.286284 18131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:46.286594 18131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:46.286798 18131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:46.294438 18131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:46.294567 18131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:46.294829 18131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.65
I20260806 02:49:46.305640 18131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:34643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.65:41307
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.65
--webserver_port=37947
--tserver_master_addrs=127.1.210.126:32935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.65
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:49:46.307161 18131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:46.308634 18131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:46.325366 18156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:46.325353 18157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:46.325487 18159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:46.327224 18131 server_base.cc:1034] running on GCE node
I20260806 02:49:46.328037 18131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:46.329097 18131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:46.330386 18131 hybrid_clock.cc:648] HybridClock initialized: now 1785984586330299 us; error 69 us; skew 500 ppm
I20260806 02:49:46.330853 18131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:49:46 dist-test-slave-8tdl krb5kdc[17499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984586, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:46.334539 18131 init.cc:377] Logged in from keytab as kudu/127.1.210.65@KRBTEST.COM (short username kudu)
I20260806 02:49:46.336460 18131 webserver.cc:460] Webserver started at http://127.1.210.65:37947/ using document root <none> and password file <none>
I20260806 02:49:46.337222 18131 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:46.337345 18131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:46.343508 18131 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 02:49:46.346581 18166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:46.348165 18131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:49:46.348341 18131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/wal
uuid: "cffa79db50164fc6a5319c74034efccf"
format_stamp: "Formatted at 2026-08-06 02:49:22 on dist-test-slave-8tdl"
server_key: "effd7f0c632917196d71b0d09b756c1e"
server_key_iv: "b263af22d2af1434d8e9bef3b3309654"
server_key_version: "encryptionkey@0"
I20260806 02:49:46.349397 18131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:46.367074 18131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:46.371783 18131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:46.372022 18131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:46.373025 18131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:49:46.374603 18131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:49:46.374764 18131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:46.374851 18131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:49:46.374934 18131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:46.412397 18131 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.65:41307
I20260806 02:49:46.412539 18280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.65:41307 every 8 connection(s)
I20260806 02:49:46.414219 18131 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
I20260806 02:49:46.415059 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 18131
I20260806 02:49:46.415917 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.66:38745
--local_ip_for_outbound_sockets=127.1.210.66
--tserver_master_addrs=127.1.210.126:32935
--webserver_port=43723
--webserver_interface=127.1.210.66
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:34643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:49:46 dist-test-slave-8tdl krb5kdc[17499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984586, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:49:46.444792 18284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:46.418110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.65:59521)
0806 02:49:46.419190 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:46.419240 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:46.421183 (+ 1943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:46.423198 (+ 2015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:46.423207 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:46.423865 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:49:46.424475 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:46.424505 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:46.427385 (+ 2880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:46.427395 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:46.429418 (+ 2023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:46.429435 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:46.430740 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:46.430750 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:46.431078 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:46.431951 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:49:46.432032 (+ 81us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:49:46.434930 (+ 2898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:49:46.439395 (+ 4465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:46.439409 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:46.439427 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:46.440106 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:46.440651 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:46.440657 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:46.440661 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:46.440928 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:46.441794 (+ 866us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:49:46.441803 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:49:46.442275 (+ 472us) client_negotiation.cc:769] Sending connection context
0806 02:49:46.442599 (+ 324us) client_negotiation.cc:240] Negotiation successful
0806 02:49:46.443270 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":499,"threads_started":1}
I20260806 02:49:46.444792 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:46.417922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:59521 (local address 127.1.210.126:32935)
0806 02:49:46.418738 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:46.418758 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:46.419461 (+ 703us) server_negotiation.cc:407] Connection header received
0806 02:49:46.421891 (+ 2430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:46.421914 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:46.422475 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:46.423000 (+ 525us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:46.424771 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:46.426921 (+ 2150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:46.429605 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:46.430376 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:46.430657 (+ 281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:46.435336 (+ 4679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:46.435348 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:46.435367 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:46.435413 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:46.439193 (+ 3780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:46.440283 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:46.440291 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:46.440299 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:46.440491 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:46.441075 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:46.441082 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:46.441086 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:46.441622 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:46.441846 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:46.442866 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0806 02:49:46.443320 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":341,"threads_started":1}
I20260806 02:49:46.448552 18281 heartbeater.cc:347] Connected to a master server at 127.1.210.126:32935
I20260806 02:49:46.448988 18281 heartbeater.cc:464] Registering TS with master...
I20260806 02:49:46.449807 18281 heartbeater.cc:511] Master 127.1.210.126:32935 requested a full tablet report, sending...
I20260806 02:49:46.452548 18092 ts_manager.cc:194] Registered new tserver with Master: cffa79db50164fc6a5319c74034efccf (127.1.210.65:41307)
I20260806 02:49:46.454777 18092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.65@KRBTEST.COM'} at 127.1.210.65:59521
W20260806 02:49:46.776397 18285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:46.776700 18285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:46.776799 18285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:46.784341 18285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:46.784467 18285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:46.784590 18285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.66
I20260806 02:49:46.794691 18285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:34643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.66:38745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.66
--webserver_port=43723
--tserver_master_addrs=127.1.210.126:32935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.66
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:49:46.796070 18285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:46.797540 18285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:46.811529 18290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:46.811507 18293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:46.811594 18291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:46.812721 18285 server_base.cc:1034] running on GCE node
I20260806 02:49:46.813422 18285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:46.814410 18285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:46.815629 18285 hybrid_clock.cc:648] HybridClock initialized: now 1785984586815575 us; error 51 us; skew 500 ppm
I20260806 02:49:46.815929 18285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:49:46 dist-test-slave-8tdl krb5kdc[17499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984586, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:46.819352 18285 init.cc:377] Logged in from keytab as kudu/127.1.210.66@KRBTEST.COM (short username kudu)
I20260806 02:49:46.821051 18285 webserver.cc:460] Webserver started at http://127.1.210.66:43723/ using document root <none> and password file <none>
I20260806 02:49:46.821748 18285 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:46.821867 18285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:46.827885 18285 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:49:46.831318 18300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:46.832858 18285 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 02:49:46.833062 18285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/wal
uuid: "eca26f9a66a946a18f4afe233ed0bbec"
format_stamp: "Formatted at 2026-08-06 02:49:23 on dist-test-slave-8tdl"
server_key: "14f7ecf0bce6698a8451bcfe232a07cc"
server_key_iv: "bb85b01eef9fcbf41ea1a15cb98805ec"
server_key_version: "encryptionkey@0"
I20260806 02:49:46.834117 18285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:46.861232 18285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:46.867234 18285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:46.867549 18285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:46.868940 18285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:49:46.870769 18285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:49:46.870898 18285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:46.871025 18285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:49:46.871097 18285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:46.911260 18285 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.66:38745
I20260806 02:49:46.911387 18413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.66:38745 every 8 connection(s)
I20260806 02:49:46.912890 18285 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
I20260806 02:49:46.920230 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 18285
I20260806 02:49:46.921058 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.67:38123
--local_ip_for_outbound_sockets=127.1.210.67
--tserver_master_addrs=127.1.210.126:32935
--webserver_port=37893
--webserver_interface=127.1.210.67
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:34643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:49:46 dist-test-slave-8tdl krb5kdc[17499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984586, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:49:46.939260 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:46.916001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:56557 (local address 127.1.210.126:32935)
0806 02:49:46.916223 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:46.916227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:46.917455 (+ 1228us) server_negotiation.cc:407] Connection header received
0806 02:49:46.919704 (+ 2249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:46.919709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:46.919837 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:46.919927 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:46.922166 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:46.922995 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:46.925549 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:46.926029 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:46.926154 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:46.931171 (+ 5017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:46.931177 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:46.931181 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:46.931219 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:46.934782 (+ 3563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:46.935942 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:46.935947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:46.935950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:46.936172 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:46.936900 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:46.936905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:46.936907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:46.937309 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:46.937441 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:46.938524 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0806 02:49:46.938869 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 02:49:46.940718 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:46.916497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.66:56557)
0806 02:49:46.917260 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:46.917308 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:46.919377 (+ 2069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:46.920249 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:46.920264 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:46.921073 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:49:46.921955 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:46.921985 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:46.923158 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:46.923166 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:46.925356 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:46.925375 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:46.926298 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:46.926309 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:46.926661 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:46.927484 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:49:46.927522 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:49:46.930836 (+ 3314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:49:46.935044 (+ 4208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:46.935060 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:46.935088 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:46.935799 (+ 711us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:46.936303 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:46.936310 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:46.936317 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:46.936605 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:46.937472 (+ 867us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:49:46.937484 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:49:46.938028 (+ 544us) client_negotiation.cc:769] Sending connection context
0806 02:49:46.938413 (+ 385us) client_negotiation.cc:240] Negotiation successful
0806 02:49:46.939117 (+ 704us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":290,"threads_started":1}
I20260806 02:49:46.942507 18415 heartbeater.cc:347] Connected to a master server at 127.1.210.126:32935
I20260806 02:49:46.942891 18415 heartbeater.cc:464] Registering TS with master...
I20260806 02:49:46.943686 18415 heartbeater.cc:511] Master 127.1.210.126:32935 requested a full tablet report, sending...
I20260806 02:49:46.945526 18092 ts_manager.cc:194] Registered new tserver with Master: eca26f9a66a946a18f4afe233ed0bbec (127.1.210.66:38745)
I20260806 02:49:46.946936 18092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.66@KRBTEST.COM'} at 127.1.210.66:56557
W20260806 02:49:47.290531 18418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:47.290940 18418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:47.291112 18418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:47.298939 18418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:47.299064 18418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:47.299181 18418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.67
I20260806 02:49:47.310051 18418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:34643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.67:38123
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.67
--webserver_port=37893
--tserver_master_addrs=127.1.210.126:32935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.67
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:49:47.311566 18418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:47.312958 18418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:47.328766 18426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:47.328785 18424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:47.329196 18423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:47.332542 18418 server_base.cc:1034] running on GCE node
I20260806 02:49:47.333336 18418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:47.334534 18418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:47.335810 18418 hybrid_clock.cc:648] HybridClock initialized: now 1785984587335791 us; error 142 us; skew 500 ppm
I20260806 02:49:47.336182 18418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:49:47 dist-test-slave-8tdl krb5kdc[17499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984587, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:47.339851 18418 init.cc:377] Logged in from keytab as kudu/127.1.210.67@KRBTEST.COM (short username kudu)
I20260806 02:49:47.341463 18418 webserver.cc:460] Webserver started at http://127.1.210.67:37893/ using document root <none> and password file <none>
I20260806 02:49:47.342213 18418 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:47.342332 18418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:47.348289 18418 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:49:47.351413 18433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:47.353047 18418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:49:47.353260 18418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/wal
uuid: "190f26f416b74325b4abedc2cffb5422"
format_stamp: "Formatted at 2026-08-06 02:49:24 on dist-test-slave-8tdl"
server_key: "787e8043adbd31e3822ddb8c340b7fde"
server_key_iv: "a67518e645cea3b84b1d1da42711a18a"
server_key_version: "encryptionkey@0"
I20260806 02:49:47.354493 18418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:47.390367 18418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:47.395082 18418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:47.395390 18418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:47.396486 18418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:49:47.397957 18418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:49:47.398095 18418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:47.398247 18418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:49:47.398361 18418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:47.437815 18418 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.67:38123
I20260806 02:49:47.437980 18545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.67:38123 every 8 connection(s)
I20260806 02:49:47.439502 18418 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
I20260806 02:49:47.449703 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 18418
Aug 06 02:49:47 dist-test-slave-8tdl krb5kdc[17499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984587, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:49:47.459506 18281 heartbeater.cc:503] Master 127.1.210.126:32935 was elected leader, sending a full tablet report...
I20260806 02:49:47.467051 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:47.442490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:45621 (local address 127.1.210.126:32935)
0806 02:49:47.443060 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:47.443064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:47.444189 (+ 1125us) server_negotiation.cc:407] Connection header received
0806 02:49:47.446144 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:47.446150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:47.446250 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:47.446379 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:47.448113 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:47.449100 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:47.451735 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:47.452475 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:47.452639 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:47.457995 (+ 5356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:47.458002 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:47.458009 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:47.458160 (+ 151us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:47.462372 (+ 4212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:47.463609 (+ 1237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:47.463613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:47.463616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:47.463751 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:47.464374 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:47.464378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:47.464380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:47.464723 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:47.464852 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:47.465950 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0806 02:49:47.466416 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"spinlock_wait_cycles":1408,"thread_start_us":322,"threads_started":1}
I20260806 02:49:47.467794 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:47.443194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.67:45621)
0806 02:49:47.443954 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:47.444049 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:47.445816 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:47.446646 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:47.446656 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:47.447301 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:49:47.447908 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:47.447927 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:47.449317 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:47.449321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:47.451190 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:47.451213 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:47.453004 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:47.453013 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:47.453524 (+ 511us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:47.454515 (+ 991us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:49:47.454555 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:49:47.457636 (+ 3081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:49:47.462587 (+ 4951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:47.462597 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:47.462621 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:47.463443 (+ 822us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:47.463884 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:47.463890 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:47.463894 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:47.464201 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:47.464884 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:49:47.464895 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:49:47.465433 (+ 538us) client_negotiation.cc:769] Sending connection context
0806 02:49:47.465770 (+ 337us) client_negotiation.cc:240] Negotiation successful
0806 02:49:47.466343 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":300,"threads_started":1}
I20260806 02:49:47.470008 18546 heartbeater.cc:347] Connected to a master server at 127.1.210.126:32935
I20260806 02:49:47.470381 18546 heartbeater.cc:464] Registering TS with master...
I20260806 02:49:47.471275 18546 heartbeater.cc:511] Master 127.1.210.126:32935 requested a full tablet report, sending...
I20260806 02:49:47.473061 18092 ts_manager.cc:194] Registered new tserver with Master: 190f26f416b74325b4abedc2cffb5422 (127.1.210.67:38123)
I20260806 02:49:47.473377 18550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:47.450831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49846 (local address 127.1.210.126:32935)
0806 02:49:47.451517 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:47.451523 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:47.452147 (+ 624us) server_negotiation.cc:407] Connection header received
0806 02:49:47.452222 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:47.452227 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:47.452315 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:47.452426 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:47.454818 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:47.456151 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:47.461199 (+ 5048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:47.461862 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:47.462014 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:47.465798 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:47.465805 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:47.465810 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:47.465870 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:47.470046 (+ 4176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:47.471017 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:47.471023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:47.471026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:47.471164 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:47.471623 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:47.471630 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:47.471633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:47.471925 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:47.472201 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:47.472660 (+ 459us) server_negotiation.cc:299] Negotiation successful
0806 02:49:47.472904 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":351,"threads_started":1}
I20260806 02:49:47.474210 18092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.67@KRBTEST.COM'} at 127.1.210.67:45621
I20260806 02:49:47.476516 1865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:49:47.484197 18550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:47.477648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49848 (local address 127.1.210.126:32935)
0806 02:49:47.477893 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:47.477902 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:47.478278 (+ 376us) server_negotiation.cc:407] Connection header received
0806 02:49:47.478494 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:47.478498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:47.478671 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:47.478863 (+ 192us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:49:47.479516 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:47.480321 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:47.481946 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:47.482424 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:47.482584 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:47.483368 (+ 784us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:49:47.483594 (+ 226us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0806 02:49:47.483972 (+ 378us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49848: token expired
Metrics: {"server-negotiator.queue_time_us":69}
W20260806 02:49:47.484395 18550 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49848: token expired
W20260806 02:49:47.485190 18014 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.210.126:32935)
I20260806 02:49:47.500514 18550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:47.485751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49856 (local address 127.1.210.126:32935)
0806 02:49:47.485962 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:47.485968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:47.485986 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:49:47.486236 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:47.486245 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:47.486306 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:47.486413 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:47.487973 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:47.488700 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:47.490360 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:47.491331 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:47.491457 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:47.494780 (+ 3323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:47.494788 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:47.494793 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:47.494842 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:47.497659 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:47.498451 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:47.498456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:47.498458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:47.498615 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:47.499053 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:47.499056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:47.499058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:47.499449 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:47.499645 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:47.499980 (+ 335us) server_negotiation.cc:299] Negotiation successful
0806 02:49:47.500208 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:49:47.504411 18092 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:49:47.512132 18092 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:49:47.551501 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:47.525534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:38745 (local address 127.0.0.1:60304)
0806 02:49:47.540380 (+ 14846us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:47.540452 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:47.540715 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:47.543148 (+ 2433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:47.543160 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:47.543270 (+ 110us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:47.543881 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:47.543899 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:47.545933 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:47.545941 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:47.548838 (+ 2897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:47.548854 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:47.550357 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:47.550363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:47.550573 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:47.550626 (+ 53us) client_negotiation.cc:769] Sending connection context
0806 02:49:47.550819 (+ 193us) client_negotiation.cc:240] Negotiation successful
0806 02:49:47.551014 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12890,"mutex_wait_us":76,"spinlock_wait_cycles":33536,"thread_start_us":13159,"threads_started":1}
I20260806 02:49:47.552170 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:47.525770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60304 (local address 127.1.210.66:38745)
0806 02:49:47.541917 (+ 16147us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:47.541929 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:47.541968 (+ 39us) server_negotiation.cc:407] Connection header received
0806 02:49:47.542090 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:47.542101 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:47.542408 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:47.542669 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:47.544176 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:47.545766 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:47.549032 (+ 3266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:47.550117 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:47.550284 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:47.550530 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:47.550882 (+ 352us) server_negotiation.cc:299] Negotiation successful
0806 02:49:47.551055 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15598,"thread_start_us":15837,"threads_started":1}
I20260806 02:49:47.552994 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:47.525730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34234 (local address 127.1.210.65:41307)
0806 02:49:47.541544 (+ 15814us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:47.541557 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:47.541652 (+ 95us) server_negotiation.cc:407] Connection header received
0806 02:49:47.541797 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:47.541810 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:47.542180 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:47.542438 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:47.545280 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:47.546662 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:47.549363 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:47.550924 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:47.551061 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:47.551299 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:47.552412 (+ 1113us) server_negotiation.cc:299] Negotiation successful
0806 02:49:47.552588 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15546,"thread_start_us":17517,"threads_started":1}
I20260806 02:49:47.553211 18562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:47.539914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:38123 (local address 127.0.0.1:57222)
0806 02:49:47.544456 (+ 4542us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:47.544475 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:47.544606 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:47.545497 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:47.545504 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:47.545529 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:47.545858 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:47.545869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:47.547945 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:47.547951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:47.550153 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:47.550168 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:47.552344 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:47.552350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:47.552541 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:47.552573 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 02:49:47.552691 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 02:49:47.552827 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4351,"mutex_wait_us":36,"thread_start_us":4139,"threads_started":1}
I20260806 02:49:47.553330 18563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:47.540348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57222 (local address 127.1.210.67:38123)
0806 02:49:47.544036 (+ 3688us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:47.544047 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:47.544739 (+ 692us) server_negotiation.cc:407] Connection header received
0806 02:49:47.544851 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:47.544860 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:47.545106 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:47.545326 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:47.546089 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:47.547757 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:47.550352 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:47.551829 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:47.551943 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:47.552166 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:47.552768 (+ 602us) server_negotiation.cc:299] Negotiation successful
0806 02:49:47.552927 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3503,"thread_start_us":3339,"threads_started":1}
I20260806 02:49:47.553501 18561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:47.525546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:41307 (local address 127.0.0.1:34234)
0806 02:49:47.540398 (+ 14852us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:47.540451 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:47.540718 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:47.543149 (+ 2431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:47.543161 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:47.544674 (+ 1513us) spinlock_profiling.cc:243] Waited 915 us on lock 0x61200002cc28. stack: 00007f271b0f766e 00007f271b0f7430 00007f271a17e5ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007f271b0ed0fd 00007f271b0ed977 00007f272f49ba7d 00007f272f49ba0e 00007f272dfc2b8d 00007f2721afe8e9 00007f2721afae45 00007f2721b82dca 00007f2721b80f68 00007f2721bb482f
0806 02:49:47.544689 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:47.545068 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:47.545084 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:47.546949 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:47.546954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:47.549183 (+ 2229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:47.549197 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:47.551415 (+ 2218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:47.551420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:47.551599 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:47.551633 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 02:49:47.551735 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 02:49:47.551869 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14708,"mutex_wait_us":75,"spinlock_wait_cycles":2129920,"thread_start_us":17225,"threads_started":1}
I20260806 02:49:47.556360 18348 tablet_service.cc:1467] Processing CreateTablet for tablet 53f3239c20a943658c00cee68fc56f68 (DEFAULT_TABLE table=test-table [id=3f25b5b10d874526920ce4a5d1b63d51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:49:47.557768 18215 tablet_service.cc:1467] Processing CreateTablet for tablet 53f3239c20a943658c00cee68fc56f68 (DEFAULT_TABLE table=test-table [id=3f25b5b10d874526920ce4a5d1b63d51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:49:47.558425 18481 tablet_service.cc:1467] Processing CreateTablet for tablet 53f3239c20a943658c00cee68fc56f68 (DEFAULT_TABLE table=test-table [id=3f25b5b10d874526920ce4a5d1b63d51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:49:47.559340 18348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53f3239c20a943658c00cee68fc56f68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:47.559571 18215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53f3239c20a943658c00cee68fc56f68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:47.562567 18481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53f3239c20a943658c00cee68fc56f68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:47.578431 18565 tablet_bootstrap.cc:492] T 53f3239c20a943658c00cee68fc56f68 P eca26f9a66a946a18f4afe233ed0bbec: Bootstrap starting.
I20260806 02:49:47.584830 18564 tablet_bootstrap.cc:492] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf: Bootstrap starting.
I20260806 02:49:47.586297 18566 tablet_bootstrap.cc:492] T 53f3239c20a943658c00cee68fc56f68 P 190f26f416b74325b4abedc2cffb5422: Bootstrap starting.
I20260806 02:49:47.589300 18564 tablet_bootstrap.cc:654] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:47.590368 18566 tablet_bootstrap.cc:654] T 53f3239c20a943658c00cee68fc56f68 P 190f26f416b74325b4abedc2cffb5422: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:47.590802 18564 log.cc:824] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:47.591858 18566 log.cc:824] T 53f3239c20a943658c00cee68fc56f68 P 190f26f416b74325b4abedc2cffb5422: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:47.591998 18565 tablet_bootstrap.cc:654] T 53f3239c20a943658c00cee68fc56f68 P eca26f9a66a946a18f4afe233ed0bbec: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:47.593462 18565 log.cc:824] T 53f3239c20a943658c00cee68fc56f68 P eca26f9a66a946a18f4afe233ed0bbec: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:47.595438 18566 tablet_bootstrap.cc:492] T 53f3239c20a943658c00cee68fc56f68 P 190f26f416b74325b4abedc2cffb5422: No bootstrap required, opened a new log
I20260806 02:49:47.595731 18566 ts_tablet_manager.cc:1397] T 53f3239c20a943658c00cee68fc56f68 P 190f26f416b74325b4abedc2cffb5422: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20260806 02:49:47.597158 18565 tablet_bootstrap.cc:492] T 53f3239c20a943658c00cee68fc56f68 P eca26f9a66a946a18f4afe233ed0bbec: No bootstrap required, opened a new log
I20260806 02:49:47.597494 18565 ts_tablet_manager.cc:1397] T 53f3239c20a943658c00cee68fc56f68 P eca26f9a66a946a18f4afe233ed0bbec: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.000s
I20260806 02:49:47.604265 18566 raft_consensus.cc:348] T 53f3239c20a943658c00cee68fc56f68 P 190f26f416b74325b4abedc2cffb5422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cffa79db50164fc6a5319c74034efccf" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 41307 } } peers { permanent_uuid: "eca26f9a66a946a18f4afe233ed0bbec" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 38745 } } peers { permanent_uuid: "190f26f416b74325b4abedc2cffb5422" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38123 } }
I20260806 02:49:47.604733 18564 tablet_bootstrap.cc:492] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf: No bootstrap required, opened a new log
I20260806 02:49:47.604766 18566 raft_consensus.cc:374] T 53f3239c20a943658c00cee68fc56f68 P 190f26f416b74325b4abedc2cffb5422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:47.604867 18566 raft_consensus.cc:729] T 53f3239c20a943658c00cee68fc56f68 P 190f26f416b74325b4abedc2cffb5422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 190f26f416b74325b4abedc2cffb5422, State: Initialized, Role: FOLLOWER
I20260806 02:49:47.605074 18564 ts_tablet_manager.cc:1397] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.004s
I20260806 02:49:47.605563 18566 consensus_queue.cc:260] T 53f3239c20a943658c00cee68fc56f68 P 190f26f416b74325b4abedc2cffb5422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cffa79db50164fc6a5319c74034efccf" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 41307 } } peers { permanent_uuid: "eca26f9a66a946a18f4afe233ed0bbec" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 38745 } } peers { permanent_uuid: "190f26f416b74325b4abedc2cffb5422" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38123 } }
I20260806 02:49:47.605832 18565 raft_consensus.cc:348] T 53f3239c20a943658c00cee68fc56f68 P eca26f9a66a946a18f4afe233ed0bbec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cffa79db50164fc6a5319c74034efccf" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 41307 } } peers { permanent_uuid: "eca26f9a66a946a18f4afe233ed0bbec" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 38745 } } peers { permanent_uuid: "190f26f416b74325b4abedc2cffb5422" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38123 } }
I20260806 02:49:47.606379 18565 raft_consensus.cc:374] T 53f3239c20a943658c00cee68fc56f68 P eca26f9a66a946a18f4afe233ed0bbec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:47.606487 18565 raft_consensus.cc:729] T 53f3239c20a943658c00cee68fc56f68 P eca26f9a66a946a18f4afe233ed0bbec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eca26f9a66a946a18f4afe233ed0bbec, State: Initialized, Role: FOLLOWER
I20260806 02:49:47.607391 18546 heartbeater.cc:503] Master 127.1.210.126:32935 was elected leader, sending a full tablet report...
I20260806 02:49:47.607235 18565 consensus_queue.cc:260] T 53f3239c20a943658c00cee68fc56f68 P eca26f9a66a946a18f4afe233ed0bbec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cffa79db50164fc6a5319c74034efccf" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 41307 } } peers { permanent_uuid: "eca26f9a66a946a18f4afe233ed0bbec" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 38745 } } peers { permanent_uuid: "190f26f416b74325b4abedc2cffb5422" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38123 } }
I20260806 02:49:47.608366 18415 heartbeater.cc:503] Master 127.1.210.126:32935 was elected leader, sending a full tablet report...
I20260806 02:49:47.609087 18565 ts_tablet_manager.cc:1428] T 53f3239c20a943658c00cee68fc56f68 P eca26f9a66a946a18f4afe233ed0bbec: Time spent starting tablet: real 0.011s user 0.013s sys 0.000s
I20260806 02:49:47.609809 18566 ts_tablet_manager.cc:1428] T 53f3239c20a943658c00cee68fc56f68 P 190f26f416b74325b4abedc2cffb5422: Time spent starting tablet: real 0.014s user 0.011s sys 0.001s
I20260806 02:49:47.615828 18564 raft_consensus.cc:348] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cffa79db50164fc6a5319c74034efccf" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 41307 } } peers { permanent_uuid: "eca26f9a66a946a18f4afe233ed0bbec" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 38745 } } peers { permanent_uuid: "190f26f416b74325b4abedc2cffb5422" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38123 } }
I20260806 02:49:47.616223 18564 raft_consensus.cc:374] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:47.616303 18564 raft_consensus.cc:729] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cffa79db50164fc6a5319c74034efccf, State: Initialized, Role: FOLLOWER
I20260806 02:49:47.616961 18564 consensus_queue.cc:260] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cffa79db50164fc6a5319c74034efccf" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 41307 } } peers { permanent_uuid: "eca26f9a66a946a18f4afe233ed0bbec" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 38745 } } peers { permanent_uuid: "190f26f416b74325b4abedc2cffb5422" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38123 } }
I20260806 02:49:47.619048 18564 ts_tablet_manager.cc:1428] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf: Time spent starting tablet: real 0.014s user 0.008s sys 0.007s
I20260806 02:49:47.622251 18571 raft_consensus.cc:482] T 53f3239c20a943658c00cee68fc56f68 P eca26f9a66a946a18f4afe233ed0bbec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:49:47.622614 18571 raft_consensus.cc:504] T 53f3239c20a943658c00cee68fc56f68 P eca26f9a66a946a18f4afe233ed0bbec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cffa79db50164fc6a5319c74034efccf" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 41307 } } peers { permanent_uuid: "eca26f9a66a946a18f4afe233ed0bbec" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 38745 } } peers { permanent_uuid: "190f26f416b74325b4abedc2cffb5422" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38123 } }
I20260806 02:49:47.624728 18571 leader_election.cc:288] T 53f3239c20a943658c00cee68fc56f68 P eca26f9a66a946a18f4afe233ed0bbec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cffa79db50164fc6a5319c74034efccf (127.1.210.65:41307), 190f26f416b74325b4abedc2cffb5422 (127.1.210.67:38123)
I20260806 02:49:47.632551 18563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:47.625630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:35285 (local address 127.1.210.67:38123)
0806 02:49:47.625876 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:47.625882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:47.625903 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:49:47.626061 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:47.626067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:47.626141 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:47.626245 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:47.627017 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:47.628269 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:47.630368 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:47.631458 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:47.631601 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:47.631658 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:47.632108 (+ 450us) server_negotiation.cc:299] Negotiation successful
0806 02:49:47.632258 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:49:47.632587 18573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:47.625065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:38123 (local address 127.1.210.66:35285)
0806 02:49:47.625775 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:47.625792 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:47.625899 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:47.626265 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:47.626270 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:47.626298 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:47.626760 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:47.626775 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:47.628425 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:47.628430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:47.630181 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:47.630194 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:47.631699 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:47.631706 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:47.631897 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:47.631941 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 02:49:47.632057 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 02:49:47.632216 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":339,"threads_started":1}
I20260806 02:49:47.633105 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:47.625171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:49665 (local address 127.1.210.65:41307)
0806 02:49:47.625370 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:47.625379 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:47.626527 (+ 1148us) server_negotiation.cc:407] Connection header received
0806 02:49:47.626665 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:47.626708 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:47.626787 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:47.626878 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:47.627699 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:47.628664 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:47.630877 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:47.631830 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:47.632014 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:47.632107 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:47.632606 (+ 499us) server_negotiation.cc:299] Negotiation successful
0806 02:49:47.632851 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 02:49:47.633497 18574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:47.625099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:41307 (local address 127.1.210.66:49665)
0806 02:49:47.626407 (+ 1308us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:47.626426 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:47.626540 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:47.626957 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:47.626962 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:47.626993 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:47.627506 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:47.627521 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:47.628823 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:47.628828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:47.630705 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:47.630721 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:47.632116 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:47.632123 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:47.632351 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:47.632436 (+ 85us) client_negotiation.cc:769] Sending connection context
0806 02:49:47.632555 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 02:49:47.632715 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1149,"thread_start_us":967,"threads_started":1}
I20260806 02:49:47.634167 18501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53f3239c20a943658c00cee68fc56f68" candidate_uuid: "eca26f9a66a946a18f4afe233ed0bbec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "190f26f416b74325b4abedc2cffb5422" is_pre_election: true
I20260806 02:49:47.634639 18235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53f3239c20a943658c00cee68fc56f68" candidate_uuid: "eca26f9a66a946a18f4afe233ed0bbec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cffa79db50164fc6a5319c74034efccf" is_pre_election: true
I20260806 02:49:47.634826 18501 raft_consensus.cc:2454] T 53f3239c20a943658c00cee68fc56f68 P 190f26f416b74325b4abedc2cffb5422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eca26f9a66a946a18f4afe233ed0bbec in term 0.
I20260806 02:49:47.635205 18235 raft_consensus.cc:2454] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eca26f9a66a946a18f4afe233ed0bbec in term 0.
I20260806 02:49:47.635757 18304 leader_election.cc:302] T 53f3239c20a943658c00cee68fc56f68 P eca26f9a66a946a18f4afe233ed0bbec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 190f26f416b74325b4abedc2cffb5422, eca26f9a66a946a18f4afe233ed0bbec; no voters:
I20260806 02:49:47.636303 18571 raft_consensus.cc:2781] T 53f3239c20a943658c00cee68fc56f68 P eca26f9a66a946a18f4afe233ed0bbec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:49:47.636458 18571 raft_consensus.cc:482] T 53f3239c20a943658c00cee68fc56f68 P eca26f9a66a946a18f4afe233ed0bbec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:49:47.636535 18571 raft_consensus.cc:3037] T 53f3239c20a943658c00cee68fc56f68 P eca26f9a66a946a18f4afe233ed0bbec [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:47.639123 18571 raft_consensus.cc:504] T 53f3239c20a943658c00cee68fc56f68 P eca26f9a66a946a18f4afe233ed0bbec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cffa79db50164fc6a5319c74034efccf" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 41307 } } peers { permanent_uuid: "eca26f9a66a946a18f4afe233ed0bbec" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 38745 } } peers { permanent_uuid: "190f26f416b74325b4abedc2cffb5422" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38123 } }
I20260806 02:49:47.639950 18571 leader_election.cc:288] T 53f3239c20a943658c00cee68fc56f68 P eca26f9a66a946a18f4afe233ed0bbec [CANDIDATE]: Term 1 election: Requested vote from peers cffa79db50164fc6a5319c74034efccf (127.1.210.65:41307), 190f26f416b74325b4abedc2cffb5422 (127.1.210.67:38123)
I20260806 02:49:47.640520 18501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53f3239c20a943658c00cee68fc56f68" candidate_uuid: "eca26f9a66a946a18f4afe233ed0bbec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "190f26f416b74325b4abedc2cffb5422"
I20260806 02:49:47.640763 18501 raft_consensus.cc:3037] T 53f3239c20a943658c00cee68fc56f68 P 190f26f416b74325b4abedc2cffb5422 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:47.640884 18235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53f3239c20a943658c00cee68fc56f68" candidate_uuid: "eca26f9a66a946a18f4afe233ed0bbec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cffa79db50164fc6a5319c74034efccf"
I20260806 02:49:47.641135 18235 raft_consensus.cc:3037] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:47.644425 18235 raft_consensus.cc:2454] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eca26f9a66a946a18f4afe233ed0bbec in term 1.
I20260806 02:49:47.644434 18501 raft_consensus.cc:2454] T 53f3239c20a943658c00cee68fc56f68 P 190f26f416b74325b4abedc2cffb5422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eca26f9a66a946a18f4afe233ed0bbec in term 1.
I20260806 02:49:47.645040 18304 leader_election.cc:302] T 53f3239c20a943658c00cee68fc56f68 P eca26f9a66a946a18f4afe233ed0bbec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 190f26f416b74325b4abedc2cffb5422, eca26f9a66a946a18f4afe233ed0bbec; no voters:
I20260806 02:49:47.645632 18571 raft_consensus.cc:2781] T 53f3239c20a943658c00cee68fc56f68 P eca26f9a66a946a18f4afe233ed0bbec [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:49:47.646271 18571 raft_consensus.cc:686] T 53f3239c20a943658c00cee68fc56f68 P eca26f9a66a946a18f4afe233ed0bbec [term 1 LEADER]: Becoming Leader. State: Replica: eca26f9a66a946a18f4afe233ed0bbec, State: Running, Role: LEADER
I20260806 02:49:47.646907 18571 consensus_queue.cc:237] T 53f3239c20a943658c00cee68fc56f68 P eca26f9a66a946a18f4afe233ed0bbec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cffa79db50164fc6a5319c74034efccf" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 41307 } } peers { permanent_uuid: "eca26f9a66a946a18f4afe233ed0bbec" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 38745 } } peers { permanent_uuid: "190f26f416b74325b4abedc2cffb5422" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38123 } }
I20260806 02:49:47.652484 18090 catalog_manager.cc:5495] T 53f3239c20a943658c00cee68fc56f68 P eca26f9a66a946a18f4afe233ed0bbec reported cstate change: term changed from 0 to 1, leader changed from <none> to eca26f9a66a946a18f4afe233ed0bbec (127.1.210.66). New cstate: current_term: 1 leader_uuid: "eca26f9a66a946a18f4afe233ed0bbec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cffa79db50164fc6a5319c74034efccf" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 41307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eca26f9a66a946a18f4afe233ed0bbec" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "190f26f416b74325b4abedc2cffb5422" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38123 } health_report { overall_health: UNKNOWN } } }
W20260806 02:49:47.665096 18416 tablet.cc:2369] T 53f3239c20a943658c00cee68fc56f68 P eca26f9a66a946a18f4afe233ed0bbec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:49:47.667124 18282 tablet.cc:2369] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:49:47.674371 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:47.668484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60314 (local address 127.1.210.66:38745)
0806 02:49:47.668688 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:47.668693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:47.668713 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:49:47.668807 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:47.668813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:47.668907 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:47.669053 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:49:47.669524 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:47.670315 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:47.671982 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:47.672455 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:47.672562 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:47.673238 (+ 676us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:49:47.673759 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:49:47.673884 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:47.674051 (+ 167us) server_negotiation.cc:299] Negotiation successful
0806 02:49:47.674129 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:49:47.683554 18501 raft_consensus.cc:1261] T 53f3239c20a943658c00cee68fc56f68 P 190f26f416b74325b4abedc2cffb5422 [term 1 FOLLOWER]: Refusing update from remote peer eca26f9a66a946a18f4afe233ed0bbec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:49:47.683554 18235 raft_consensus.cc:1261] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [term 1 FOLLOWER]: Refusing update from remote peer eca26f9a66a946a18f4afe233ed0bbec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:49:47.684767 18571 consensus_queue.cc:1035] T 53f3239c20a943658c00cee68fc56f68 P eca26f9a66a946a18f4afe233ed0bbec [LEADER]: Connected to new peer: Peer: permanent_uuid: "cffa79db50164fc6a5319c74034efccf" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 41307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:49:47.685181 18575 consensus_queue.cc:1035] T 53f3239c20a943658c00cee68fc56f68 P eca26f9a66a946a18f4afe233ed0bbec [LEADER]: Connected to new peer: Peer: permanent_uuid: "190f26f416b74325b4abedc2cffb5422" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 02:49:47.691583 18547 tablet.cc:2369] T 53f3239c20a943658c00cee68fc56f68 P 190f26f416b74325b4abedc2cffb5422: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:49:47.696579 18578 mvcc.cc:205] Tried to move back new op lower bound from 7315392871139868672 to 7315392871006666752. Current Snapshot: MvccSnapshot[applied={T|T < 7315392871139868672}]
I20260806 02:49:47.713809 18090 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:49:47.714286 18090 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:49:47.718197 18090 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c: Sending DeleteTablet for 3 replicas of tablet 53f3239c20a943658c00cee68fc56f68
I20260806 02:49:47.719498 18215 tablet_service.cc:1514] Processing DeleteTablet for tablet 53f3239c20a943658c00cee68fc56f68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:49:47 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:34234
I20260806 02:49:47.719903 18348 tablet_service.cc:1514] Processing DeleteTablet for tablet 53f3239c20a943658c00cee68fc56f68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:49:47 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:60304
I20260806 02:49:47.720436 18481 tablet_service.cc:1514] Processing DeleteTablet for tablet 53f3239c20a943658c00cee68fc56f68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:49:47 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:57222
I20260806 02:49:47.720764 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 18131
I20260806 02:49:47.745340 18590 tablet_replica.cc:330] T 53f3239c20a943658c00cee68fc56f68 P 190f26f416b74325b4abedc2cffb5422: stopping tablet replica
I20260806 02:49:47.745939 18590 raft_consensus.cc:2229] T 53f3239c20a943658c00cee68fc56f68 P 190f26f416b74325b4abedc2cffb5422 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:49:47.746665 18590 raft_consensus.cc:2258] T 53f3239c20a943658c00cee68fc56f68 P 190f26f416b74325b4abedc2cffb5422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:49:47.750092 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:47.722512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34244 (local address 127.1.210.65:41307)
0806 02:49:47.722763 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:47.722772 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:47.722794 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:49:47.722862 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:47.722869 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:47.722962 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:47.723075 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:47.725454 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:47.726595 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:47.728769 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:47.729451 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:47.729595 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:47.738559 (+ 8964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:47.738592 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:47.738621 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:47.738720 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:47.744622 (+ 5902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:47.745967 (+ 1345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:47.745980 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:47.745984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:47.746363 (+ 379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:47.747081 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:47.747087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:47.747090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:47.747631 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:47.747977 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:47.749356 (+ 1379us) server_negotiation.cc:299] Negotiation successful
0806 02:49:47.749633 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 02:49:47.751250 18276 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:49:47.753997 18591 tablet_replica.cc:330] T 53f3239c20a943658c00cee68fc56f68 P eca26f9a66a946a18f4afe233ed0bbec: stopping tablet replica
I20260806 02:49:47.754635 18591 raft_consensus.cc:2229] T 53f3239c20a943658c00cee68fc56f68 P eca26f9a66a946a18f4afe233ed0bbec [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:49:47.755577 18591 raft_consensus.cc:2258] T 53f3239c20a943658c00cee68fc56f68 P eca26f9a66a946a18f4afe233ed0bbec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:49:47.756242 18590 ts_tablet_manager.cc:1905] T 53f3239c20a943658c00cee68fc56f68 P 190f26f416b74325b4abedc2cffb5422: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:49:47.757813 18591 ts_tablet_manager.cc:1905] T 53f3239c20a943658c00cee68fc56f68 P eca26f9a66a946a18f4afe233ed0bbec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:49:47.764472 18590 ts_tablet_manager.cc:1918] T 53f3239c20a943658c00cee68fc56f68 P 190f26f416b74325b4abedc2cffb5422: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:49:47.764792 18590 log.cc:1196] T 53f3239c20a943658c00cee68fc56f68 P 190f26f416b74325b4abedc2cffb5422: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/wal/wals/53f3239c20a943658c00cee68fc56f68
I20260806 02:49:47.765542 18590 ts_tablet_manager.cc:1939] T 53f3239c20a943658c00cee68fc56f68 P 190f26f416b74325b4abedc2cffb5422: Deleting consensus metadata
I20260806 02:49:47.766090 18591 ts_tablet_manager.cc:1918] T 53f3239c20a943658c00cee68fc56f68 P eca26f9a66a946a18f4afe233ed0bbec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:49:47.766309 18591 log.cc:1196] T 53f3239c20a943658c00cee68fc56f68 P eca26f9a66a946a18f4afe233ed0bbec: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/wal/wals/53f3239c20a943658c00cee68fc56f68
I20260806 02:49:47.766964 18591 ts_tablet_manager.cc:1939] T 53f3239c20a943658c00cee68fc56f68 P eca26f9a66a946a18f4afe233ed0bbec: Deleting consensus metadata
I20260806 02:49:47.768072 18078 catalog_manager.cc:4841] TS 190f26f416b74325b4abedc2cffb5422 (127.1.210.67:38123): tablet 53f3239c20a943658c00cee68fc56f68 (table test-table [id=3f25b5b10d874526920ce4a5d1b63d51]) successfully deleted
I20260806 02:49:47.771279 18078 catalog_manager.cc:4841] TS eca26f9a66a946a18f4afe233ed0bbec (127.1.210.66:38745): tablet 53f3239c20a943658c00cee68fc56f68 (table test-table [id=3f25b5b10d874526920ce4a5d1b63d51]) successfully deleted
I20260806 02:49:47.855805 18589 tablet_replica.cc:330] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf: stopping tablet replica
I20260806 02:49:47.856052 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 18131
W20260806 02:49:47.877967 18077 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.210.65:41307 (error 108)
I20260806 02:49:47.878507 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 18285
I20260806 02:49:47.881136 18561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:47.880632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:41307 (local address 127.0.0.1:34246)
0806 02:49:47.880828 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:47.880965 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:41307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 02:49:47.881417 18077 catalog_manager.cc:4584] TS cffa79db50164fc6a5319c74034efccf (127.1.210.65:41307): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 53f3239c20a943658c00cee68fc56f68: Network error: Client connection negotiation failed: client connection to 127.1.210.65:41307: connect: Connection refused (error 111)
I20260806 02:49:47.898382 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:47.880216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60316 (local address 127.1.210.66:38745)
0806 02:49:47.880418 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:47.880424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:47.880444 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:49:47.880521 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:47.880531 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:47.880620 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:47.880721 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:47.882590 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:47.884130 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:47.886117 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:47.886887 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:47.887056 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:47.890568 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:47.890591 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:47.890627 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:47.890705 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:47.894982 (+ 4277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:47.895896 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:47.895910 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:47.895914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:47.896120 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:47.896589 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:47.896594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:47.896597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:47.897096 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:47.897429 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:47.897711 (+ 282us) server_negotiation.cc:299] Negotiation successful
0806 02:49:47.897954 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 02:49:47.899499 18409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:49:47.935689 18561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:47.935091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:41307 (local address 127.0.0.1:34258)
0806 02:49:47.935339 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:47.935484 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:41307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260806 02:49:47.936930 18561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:47.936496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:41307 (local address 127.0.0.1:34262)
0806 02:49:47.936669 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:47.936779 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:41307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260806 02:49:47.997617 18561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:47.996989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:41307 (local address 127.0.0.1:34266)
0806 02:49:47.997273 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:47.997392 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:41307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260806 02:49:47.998916 18561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:47.998457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:41307 (local address 127.0.0.1:34280)
0806 02:49:47.998645 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:47.998759 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:41307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 02:49:48.019124 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 18285
I20260806 02:49:48.040092 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 18418
I20260806 02:49:48.056839 18563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:48.041129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57228 (local address 127.1.210.67:38123)
0806 02:49:48.041452 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:48.041459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:48.041544 (+ 85us) server_negotiation.cc:407] Connection header received
0806 02:49:48.041794 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:48.041804 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:48.041872 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:48.041994 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:48.043615 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:48.044398 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:48.045796 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:48.046222 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:48.046386 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:48.049691 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:48.049708 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:48.049725 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:48.049770 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:48.053196 (+ 3426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:48.054135 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:48.054144 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:48.054146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:48.054295 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:48.055085 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:48.055089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:48.055091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:48.055502 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:48.055829 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:48.056182 (+ 353us) server_negotiation.cc:299] Negotiation successful
0806 02:49:48.056443 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260806 02:49:48.057626 18541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:49:48.080224 18561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:48.079624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:41307 (local address 127.0.0.1:34296)
0806 02:49:48.079898 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:48.080019 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:41307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260806 02:49:48.081692 18561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:48.081107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:41307 (local address 127.0.0.1:34300)
0806 02:49:48.081372 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:48.081512 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:41307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260806 02:49:48.155272 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 18418
I20260806 02:49:48.174790 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 18058
I20260806 02:49:48.176211 18123 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:49:48.296209 18561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:48.295457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:41307 (local address 127.0.0.1:34306)
0806 02:49:48.295767 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:48.295888 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:41307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260806 02:49:48.296375 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 18058
I20260806 02:50:08.320080 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:32935
--webserver_interface=127.1.210.126
--webserver_port=43525
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:34643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:32935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:08.702991 18597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:08.703320 18597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:08.703541 18597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:08.711077 18597 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 02:50:08.711195 18597 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 02:50:08.711272 18597 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 02:50:08.711364 18597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:08.711460 18597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:08.711588 18597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:08.711669 18597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:08.711755 18597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:08.722505 18597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:34643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:32935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:32935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=43525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:50:08.724007 18597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:08.725550 18597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:08.739610 18603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:08.739637 18605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:08.739610 18602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:08.740106 18597 server_base.cc:1034] running on GCE node
I20260806 02:50:08.741431 18597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:08.743175 18597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:08.744451 18597 hybrid_clock.cc:648] HybridClock initialized: now 1785984608744384 us; error 47 us; skew 500 ppm
I20260806 02:50:08.744855 18597 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:50:08 dist-test-slave-8tdl krb5kdc[17499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984608, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:08.748401 18597 init.cc:377] Logged in from keytab as kudu/127.1.210.126@KRBTEST.COM (short username kudu)
I20260806 02:50:08.750186 18597 webserver.cc:460] Webserver started at http://127.1.210.126:43525/ using document root <none> and password file <none>
I20260806 02:50:08.751116 18597 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:08.751264 18597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:08.757936 18597 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:50:08.761896 18612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:08.763967 18597 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:50:08.764227 18597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "fab79f09beb9418f9face54c9be3c59c"
format_stamp: "Formatted at 2026-08-06 02:49:22 on dist-test-slave-8tdl"
server_key: "022c7a4947764434fbfbfef604332080"
server_key_iv: "fba9cbe23d2caedad5bacc980a2cc8dd"
server_key_version: "encryptionkey@0"
I20260806 02:50:08.765372 18597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:08.780089 18597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:08.784955 18597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:08.785291 18597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:08.806891 18597 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:32935
I20260806 02:50:08.806986 18663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:32935 every 8 connection(s)
I20260806 02:50:08.808594 18597 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:08.812542 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 18597
I20260806 02:50:08.813774 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.65:41307
--local_ip_for_outbound_sockets=127.1.210.65
--tserver_master_addrs=127.1.210.126:32935
--webserver_port=37947
--webserver_interface=127.1.210.65
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:34643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:50:08.817940 18664 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:50:08.824323 18664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c: Bootstrap starting.
I20260806 02:50:08.860698 18664 log.cc:824] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:08.885257 18664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:50:08.886133 18664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c: Bootstrap complete.
I20260806 02:50:08.902540 18664 raft_consensus.cc:348] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fab79f09beb9418f9face54c9be3c59c" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 32935 } }
I20260806 02:50:08.904028 18664 raft_consensus.cc:729] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fab79f09beb9418f9face54c9be3c59c, State: Initialized, Role: FOLLOWER
I20260806 02:50:08.904870 18664 consensus_queue.cc:260] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fab79f09beb9418f9face54c9be3c59c" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 32935 } }
I20260806 02:50:08.905185 18664 raft_consensus.cc:388] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:50:08.905331 18664 raft_consensus.cc:482] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:50:08.905521 18664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [term 2 FOLLOWER]: Advancing to term 3
I20260806 02:50:08.912891 18664 raft_consensus.cc:504] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fab79f09beb9418f9face54c9be3c59c" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 32935 } }
I20260806 02:50:08.913669 18664 leader_election.cc:302] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fab79f09beb9418f9face54c9be3c59c; no voters:
I20260806 02:50:08.914508 18664 leader_election.cc:288] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 02:50:08.914711 18668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [term 3 FOLLOWER]: Leader election won for term 3
I20260806 02:50:08.916164 18668 raft_consensus.cc:686] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [term 3 LEADER]: Becoming Leader. State: Replica: fab79f09beb9418f9face54c9be3c59c, State: Running, Role: LEADER
I20260806 02:50:08.916851 18668 consensus_queue.cc:237] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fab79f09beb9418f9face54c9be3c59c" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 32935 } }
I20260806 02:50:08.917405 18664 sys_catalog.cc:564] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:08.919687 18671 sys_catalog.cc:455] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fab79f09beb9418f9face54c9be3c59c. Latest consensus state: current_term: 3 leader_uuid: "fab79f09beb9418f9face54c9be3c59c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fab79f09beb9418f9face54c9be3c59c" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 32935 } } }
I20260806 02:50:08.920100 18671 sys_catalog.cc:458] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:08.920785 18669 sys_catalog.cc:455] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "fab79f09beb9418f9face54c9be3c59c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fab79f09beb9418f9face54c9be3c59c" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 32935 } } }
I20260806 02:50:08.921113 18669 sys_catalog.cc:458] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:08.929059 18674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:50:08.942102 18674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:08.944360 18674 catalog_manager.cc:1259] Loaded cluster ID: bcfb26b2720d4a18a8ed26ea8248af08
I20260806 02:50:08.944528 18674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:08.950344 18674 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:08.952992 18674 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c: Loaded TSK: 0
I20260806 02:50:08.953485 18674 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fab79f09beb9418f9face54c9be3c59c: Loaded TSK: 1
F20260806 02:50:08.955219 18674 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1b4b2ad9ad google::LogMessage::Fail() at ??:0
@ 0x7f1b4b2b088a google::LogMessage::SendToLog() at ??:0
@ 0x7f1b4b2ad4de google::LogMessage::Flush() at ??:0
@ 0x7f1b4b2aea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20260806 02:50:08.966518 18686 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f1b4f0117d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f1b4f00073e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f1b60cdc1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f1b60cdb783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f1b60d672a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f1b60d670c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f1b60db15e9 std::function<>::operator()() at ??:0
@ 0x7f1b60cdfcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f1b60cddc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1b60d65610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f1b60d65439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f1b4def9345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1b4defc91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f1b4defc799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f1b4ded7eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1b511b16db start_thread at ??:0
@ 0x7f1b4a14761f clone at ??:0
W20260806 02:50:09.300627 18666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:09.300940 18666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:09.301050 18666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:09.309258 18666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:09.309428 18666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:09.309574 18666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.65
I20260806 02:50:09.320521 18666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:34643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.65:41307
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.65
--webserver_port=37947
--tserver_master_addrs=127.1.210.126:32935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.65
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:50:09.321960 18666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:09.323436 18666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:09.339249 18716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:09.339344 18714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:09.339277 18713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:09.340299 18666 server_base.cc:1034] running on GCE node
I20260806 02:50:09.340992 18666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:09.342056 18666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:09.343302 18666 hybrid_clock.cc:648] HybridClock initialized: now 1785984609343250 us; error 46 us; skew 500 ppm
I20260806 02:50:09.343694 18666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:50:09 dist-test-slave-8tdl krb5kdc[17499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984609, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:09.347343 18666 init.cc:377] Logged in from keytab as kudu/127.1.210.65@KRBTEST.COM (short username kudu)
I20260806 02:50:09.349231 18666 webserver.cc:460] Webserver started at http://127.1.210.65:37947/ using document root <none> and password file <none>
I20260806 02:50:09.350013 18666 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:09.350176 18666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:09.356801 18666 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260806 02:50:09.360497 18723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:09.362170 18666 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:50:09.362406 18666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/wal
uuid: "cffa79db50164fc6a5319c74034efccf"
format_stamp: "Formatted at 2026-08-06 02:49:22 on dist-test-slave-8tdl"
server_key: "effd7f0c632917196d71b0d09b756c1e"
server_key_iv: "b263af22d2af1434d8e9bef3b3309654"
server_key_version: "encryptionkey@0"
I20260806 02:50:09.363641 18666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:09.379978 18666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:09.385617 18666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:09.385954 18666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:09.387073 18666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:09.389166 18730 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260806 02:50:09.395713 18666 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 02:50:09.395920 18666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.000s sys 0.001s
I20260806 02:50:09.396085 18666 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 02:50:09.399745 18666 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 02:50:09.399887 18666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.003s sys 0.000s
I20260806 02:50:09.400354 18730 tablet_bootstrap.cc:492] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf: Bootstrap starting.
I20260806 02:50:09.430091 18730 log.cc:824] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:09.445137 18730 tablet_bootstrap.cc:492] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:50:09.446058 18730 tablet_bootstrap.cc:492] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf: Bootstrap complete.
I20260806 02:50:09.447753 18730 ts_tablet_manager.cc:1397] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf: Time spent bootstrapping tablet: real 0.048s user 0.019s sys 0.026s
I20260806 02:50:09.448678 18666 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.65:41307
I20260806 02:50:09.448763 18837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.65:41307 every 8 connection(s)
I20260806 02:50:09.450089 18666 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
I20260806 02:50:09.455806 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 18666
I20260806 02:50:09.456000 18840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:09.453294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.65:58167)
0806 02:50:09.454280 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:09.454622 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":388,"threads_started":1}
I20260806 02:50:09.456671 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.66:38745
--local_ip_for_outbound_sockets=127.1.210.66
--tserver_master_addrs=127.1.210.126:32935
--webserver_port=43723
--webserver_interface=127.1.210.66
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:34643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:09.456943 18838 heartbeater.cc:644] Failed to heartbeat to 127.1.210.126:32935 (0 consecutive failures): Network error: Failed to ping master at 127.1.210.126:32935: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
I20260806 02:50:09.458228 18840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:09.457634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.65:49625)
0806 02:50:09.457864 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:09.457982 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260806 02:50:09.457240 18730 raft_consensus.cc:348] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cffa79db50164fc6a5319c74034efccf" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 41307 } } peers { permanent_uuid: "eca26f9a66a946a18f4afe233ed0bbec" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 38745 } } peers { permanent_uuid: "190f26f416b74325b4abedc2cffb5422" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38123 } }
I20260806 02:50:09.458540 18730 raft_consensus.cc:729] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cffa79db50164fc6a5319c74034efccf, State: Initialized, Role: FOLLOWER
I20260806 02:50:09.459199 18730 consensus_queue.cc:260] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cffa79db50164fc6a5319c74034efccf" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 41307 } } peers { permanent_uuid: "eca26f9a66a946a18f4afe233ed0bbec" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 38745 } } peers { permanent_uuid: "190f26f416b74325b4abedc2cffb5422" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38123 } }
I20260806 02:50:09.459728 18840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:09.459090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.65:34407)
0806 02:50:09.459410 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:09.459526 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198}
W20260806 02:50:09.460019 18838 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 02:50:09.461210 18730 ts_tablet_manager.cc:1428] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf: Time spent starting tablet: real 0.013s user 0.012s sys 0.001s
I20260806 02:50:09.461436 18840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:09.460943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.65:47445)
0806 02:50:09.461188 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:09.461280 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260806 02:50:09.462492 18840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:09.462015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.65:53975)
0806 02:50:09.462230 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:09.462319 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260806 02:50:09.839664 18842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:09.840001 18842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:09.840152 18842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:09.848120 18842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:09.848253 18842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:09.848484 18842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.66
I20260806 02:50:09.859464 18842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:34643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.66:38745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.66
--webserver_port=43723
--tserver_master_addrs=127.1.210.126:32935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.66
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:50:09.860972 18842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:09.862464 18842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:09.877697 18847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:09.877641 18850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:09.877672 18848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:09.878579 18842 server_base.cc:1034] running on GCE node
I20260806 02:50:09.879478 18842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:09.880770 18842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:09.882015 18842 hybrid_clock.cc:648] HybridClock initialized: now 1785984609881936 us; error 83 us; skew 500 ppm
I20260806 02:50:09.882426 18842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:50:09 dist-test-slave-8tdl krb5kdc[17499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984609, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:09.887243 18842 init.cc:377] Logged in from keytab as kudu/127.1.210.66@KRBTEST.COM (short username kudu)
I20260806 02:50:09.889458 18842 webserver.cc:460] Webserver started at http://127.1.210.66:43723/ using document root <none> and password file <none>
I20260806 02:50:09.890357 18842 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:09.890542 18842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:09.897022 18842 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:50:09.900259 18857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:09.901966 18842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:50:09.902235 18842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/wal
uuid: "eca26f9a66a946a18f4afe233ed0bbec"
format_stamp: "Formatted at 2026-08-06 02:49:23 on dist-test-slave-8tdl"
server_key: "14f7ecf0bce6698a8451bcfe232a07cc"
server_key_iv: "bb85b01eef9fcbf41ea1a15cb98805ec"
server_key_version: "encryptionkey@0"
I20260806 02:50:09.903395 18842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:09.936271 18842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:09.941279 18842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:09.941603 18842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:09.942739 18842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:09.944227 18842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:09.944355 18842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:09.944489 18842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:09.944579 18842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:09.980424 18842 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.66:38745
I20260806 02:50:09.980568 18971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.66:38745 every 8 connection(s)
I20260806 02:50:09.981993 18842 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
I20260806 02:50:09.987277 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:09.985244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.66:46343)
0806 02:50:09.985992 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:09.986288 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":299,"threads_started":1}
W20260806 02:50:09.988070 18972 heartbeater.cc:644] Failed to heartbeat to 127.1.210.126:32935 (0 consecutive failures): Network error: Failed to ping master at 127.1.210.126:32935: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
I20260806 02:50:09.989010 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:09.988641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.66:60853)
0806 02:50:09.988810 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:09.988884 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260806 02:50:09.990079 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:09.989676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.66:42193)
0806 02:50:09.989840 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:09.989965 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260806 02:50:09.990410 18972 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 02:50:09.991150 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 18842
I20260806 02:50:09.991796 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.67:38123
--local_ip_for_outbound_sockets=127.1.210.67
--tserver_master_addrs=127.1.210.126:32935
--webserver_port=37893
--webserver_interface=127.1.210.67
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:34643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:10.361164 18975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:10.361505 18975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:10.361611 18975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:10.369827 18975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:10.369978 18975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:10.370100 18975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.67
I20260806 02:50:10.380501 18975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:34643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.67:38123
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.67
--webserver_port=37893
--tserver_master_addrs=127.1.210.126:32935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.67
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:50:10.381976 18975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:10.383379 18975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:10.398592 18983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:10.398546 18980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:10.398648 18981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:10.399514 18975 server_base.cc:1034] running on GCE node
I20260806 02:50:10.400223 18975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:10.401285 18975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:10.402593 18975 hybrid_clock.cc:648] HybridClock initialized: now 1785984610402516 us; error 57 us; skew 500 ppm
I20260806 02:50:10.403103 18975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:50:10 dist-test-slave-8tdl krb5kdc[17499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984610, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:10.407004 18975 init.cc:377] Logged in from keytab as kudu/127.1.210.67@KRBTEST.COM (short username kudu)
I20260806 02:50:10.409022 18975 webserver.cc:460] Webserver started at http://127.1.210.67:37893/ using document root <none> and password file <none>
I20260806 02:50:10.409893 18975 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:10.410030 18975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:10.416692 18975 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:50:10.419967 18990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:10.421725 18975 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:50:10.421962 18975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/wal
uuid: "190f26f416b74325b4abedc2cffb5422"
format_stamp: "Formatted at 2026-08-06 02:49:24 on dist-test-slave-8tdl"
server_key: "787e8043adbd31e3822ddb8c340b7fde"
server_key_iv: "a67518e645cea3b84b1d1da42711a18a"
server_key_version: "encryptionkey@0"
I20260806 02:50:10.423143 18975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:10.441264 18975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:10.445892 18975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:10.446220 18975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:10.447454 18975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:10.449318 18975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:10.449432 18975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:10.449541 18975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:10.449623 18975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:10.464587 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:10.463559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.65:49273)
0806 02:50:10.464143 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:10.464260 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":242,"threads_started":1}
I20260806 02:50:10.489118 18975 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.67:38123
I20260806 02:50:10.489198 19105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.67:38123 every 8 connection(s)
I20260806 02:50:10.490940 18975 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
I20260806 02:50:10.494108 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 18975
I20260806 02:50:10.496791 19108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:10.494391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.67:55575)
0806 02:50:10.495409 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:10.495722 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":405,"threads_started":1}
W20260806 02:50:10.497522 19106 heartbeater.cc:644] Failed to heartbeat to 127.1.210.126:32935 (0 consecutive failures): Network error: Failed to ping master at 127.1.210.126:32935: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
I20260806 02:50:10.498492 19108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:10.498088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.67:39515)
0806 02:50:10.498278 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:10.498367 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260806 02:50:10.499518 19108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:10.499144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.67:53459)
0806 02:50:10.499312 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:10.499403 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260806 02:50:10.499742 19106 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 02:50:10.850656 19110 raft_consensus.cc:482] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:50:10.851069 19110 raft_consensus.cc:504] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cffa79db50164fc6a5319c74034efccf" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 41307 } } peers { permanent_uuid: "eca26f9a66a946a18f4afe233ed0bbec" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 38745 } } peers { permanent_uuid: "190f26f416b74325b4abedc2cffb5422" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38123 } }
I20260806 02:50:10.852886 19110 leader_election.cc:288] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eca26f9a66a946a18f4afe233ed0bbec (127.1.210.66:38745), 190f26f416b74325b4abedc2cffb5422 (127.1.210.67:38123)
Aug 06 02:50:10 dist-test-slave-8tdl krb5kdc[17499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984609, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for kudu/127.1.210.67@KRBTEST.COM
Aug 06 02:50:10 dist-test-slave-8tdl krb5kdc[17499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984609, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for kudu/127.1.210.66@KRBTEST.COM
I20260806 02:50:10.875854 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:10.853579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:38123 (local address 127.1.210.65:38103)
0806 02:50:10.855485 (+ 1906us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:10.855506 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:10.855626 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:10.858271 (+ 2645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:10.858276 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:10.859004 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:10.859380 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:10.859393 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:10.861152 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:10.861157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:10.862429 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:10.862444 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:10.863843 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:10.863850 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:10.864125 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:10.864921 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:10.864947 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:10.867969 (+ 3022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:10.871758 (+ 3789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:10.871768 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:10.871784 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:10.872266 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:10.872758 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:10.872764 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:10.872768 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:10.873087 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:10.874071 (+ 984us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:10.874079 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:10.874500 (+ 421us) client_negotiation.cc:769] Sending connection context
0806 02:50:10.874889 (+ 389us) client_negotiation.cc:240] Negotiation successful
0806 02:50:10.875463 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1742,"thread_start_us":1562,"threads_started":1}
I20260806 02:50:10.876230 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:10.854062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:38103 (local address 127.1.210.67:38123)
0806 02:50:10.855905 (+ 1843us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:10.855916 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:10.855952 (+ 36us) server_negotiation.cc:407] Connection header received
0806 02:50:10.857558 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:10.857568 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:10.857962 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:10.858319 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:10.859624 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:10.860941 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:10.862626 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:10.863513 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:10.863715 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:10.868335 (+ 4620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:10.868345 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:10.868361 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:10.868408 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:10.871581 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:10.872442 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:10.872451 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:10.872457 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:10.872631 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:10.873248 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:10.873252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:10.873254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:10.873895 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:10.874118 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:10.875269 (+ 1151us) server_negotiation.cc:299] Negotiation successful
0806 02:50:10.875792 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1673,"thread_start_us":1488,"threads_started":1}
I20260806 02:50:10.876292 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:10.853176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:38745 (local address 127.1.210.65:43219)
0806 02:50:10.853518 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:10.853585 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:10.854224 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:10.857807 (+ 3583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:10.857823 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:10.858609 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:10.859313 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:10.859337 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:10.860987 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:10.860990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:10.862398 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:10.862416 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:10.863967 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:10.863973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:10.864188 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:10.865054 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:10.865074 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:10.868425 (+ 3351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:10.872620 (+ 4195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:10.872627 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:10.872631 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:10.873077 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:10.873603 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:10.873609 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:10.873613 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:10.873776 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:10.875223 (+ 1447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:10.875228 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:10.875393 (+ 165us) client_negotiation.cc:769] Sending connection context
0806 02:50:10.875539 (+ 146us) client_negotiation.cc:240] Negotiation successful
0806 02:50:10.875803 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210}
I20260806 02:50:10.877245 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:10.853607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:43219 (local address 127.1.210.66:38745)
0806 02:50:10.855123 (+ 1516us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:10.855144 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:10.855191 (+ 47us) server_negotiation.cc:407] Connection header received
0806 02:50:10.856879 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:10.856888 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:10.857299 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:10.857692 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:10.859534 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:10.860773 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:10.862583 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:10.863713 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:10.863961 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:10.868751 (+ 4790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:10.868760 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:10.868775 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:10.868818 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:10.872447 (+ 3629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:10.873249 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:10.873257 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:10.873263 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:10.873455 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:10.873951 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:10.873959 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:10.873966 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:10.875039 (+ 1073us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:10.876025 (+ 986us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:10.876348 (+ 323us) server_negotiation.cc:299] Negotiation successful
0806 02:50:10.876822 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1329,"thread_start_us":1192,"threads_started":1}
W20260806 02:50:10.880149 18727 leader_election.cc:341] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 190f26f416b74325b4abedc2cffb5422 (127.1.210.67:38123): Not found: Tablet not found: 53f3239c20a943658c00cee68fc56f68
W20260806 02:50:10.880561 18727 leader_election.cc:341] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eca26f9a66a946a18f4afe233ed0bbec (127.1.210.66:38745): Not found: Tablet not found: 53f3239c20a943658c00cee68fc56f68
I20260806 02:50:10.880718 18727 leader_election.cc:302] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cffa79db50164fc6a5319c74034efccf; no voters: 190f26f416b74325b4abedc2cffb5422, eca26f9a66a946a18f4afe233ed0bbec
I20260806 02:50:10.881158 19110 raft_consensus.cc:2726] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:50:10.992223 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:10.991134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.66:54695)
0806 02:50:10.991736 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:10.991907 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":337,"threads_started":1}
I20260806 02:50:11.466526 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:11.465718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.65:38289)
0806 02:50:11.466266 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:11.466359 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":303,"threads_started":1}
I20260806 02:50:11.501164 19116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:11.500375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.67:40709)
0806 02:50:11.500833 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:11.500951 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":224,"threads_started":1}
I20260806 02:50:11.994374 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:11.993423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.66:59341)
0806 02:50:11.994081 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:11.994189 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":368,"threads_started":1}
I20260806 02:50:12.468713 19119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:12.467796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.65:45571)
0806 02:50:12.468412 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:12.468509 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":295,"threads_started":1}
I20260806 02:50:12.503450 19120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:12.502397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.67:38449)
0806 02:50:12.503050 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:12.503180 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":327,"threads_started":1}
I20260806 02:50:12.935103 19121 raft_consensus.cc:482] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:50:12.935451 19121 raft_consensus.cc:504] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cffa79db50164fc6a5319c74034efccf" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 41307 } } peers { permanent_uuid: "eca26f9a66a946a18f4afe233ed0bbec" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 38745 } } peers { permanent_uuid: "190f26f416b74325b4abedc2cffb5422" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38123 } }
I20260806 02:50:12.936246 19121 leader_election.cc:288] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eca26f9a66a946a18f4afe233ed0bbec (127.1.210.66:38745), 190f26f416b74325b4abedc2cffb5422 (127.1.210.67:38123)
W20260806 02:50:12.937681 18727 leader_election.cc:341] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 190f26f416b74325b4abedc2cffb5422 (127.1.210.67:38123): Not found: Tablet not found: 53f3239c20a943658c00cee68fc56f68
W20260806 02:50:12.937927 18727 leader_election.cc:341] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eca26f9a66a946a18f4afe233ed0bbec (127.1.210.66:38745): Not found: Tablet not found: 53f3239c20a943658c00cee68fc56f68
I20260806 02:50:12.938025 18727 leader_election.cc:302] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cffa79db50164fc6a5319c74034efccf; no voters: 190f26f416b74325b4abedc2cffb5422, eca26f9a66a946a18f4afe233ed0bbec
I20260806 02:50:12.938365 19121 raft_consensus.cc:2726] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:50:12.996655 19122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:12.995748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.66:58731)
0806 02:50:12.996344 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:12.996466 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":332,"threads_started":1}
I20260806 02:50:13.470711 19123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:13.469789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.65:36697)
0806 02:50:13.470366 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:13.470467 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":322,"threads_started":1}
I20260806 02:50:13.505468 19124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:13.504623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.67:37357)
0806 02:50:13.505192 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:13.505299 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":317,"threads_started":1}
I20260806 02:50:13.999086 19125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:13.997871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.66:35725)
0806 02:50:13.998608 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:13.998773 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":365,"threads_started":1}
I20260806 02:50:14.473097 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:14.472109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.65:49653)
0806 02:50:14.472769 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:14.472901 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":240,"threads_started":1}
I20260806 02:50:14.507369 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:14.506516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.67:53883)
0806 02:50:14.507089 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:14.507190 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":327,"threads_started":1}
I20260806 02:50:15.001302 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.000378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.66:55859)
0806 02:50:15.000996 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:15.001113 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":316,"threads_started":1}
I20260806 02:50:15.475106 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.474207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.65:41145)
0806 02:50:15.474791 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:15.474916 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":263,"threads_started":1}
I20260806 02:50:15.509086 19130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.508346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.67:55973)
0806 02:50:15.508800 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:15.508887 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":254,"threads_started":1}
I20260806 02:50:15.618242 19131 raft_consensus.cc:482] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:50:15.618454 19131 raft_consensus.cc:504] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cffa79db50164fc6a5319c74034efccf" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 41307 } } peers { permanent_uuid: "eca26f9a66a946a18f4afe233ed0bbec" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 38745 } } peers { permanent_uuid: "190f26f416b74325b4abedc2cffb5422" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38123 } }
I20260806 02:50:15.619318 19131 leader_election.cc:288] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eca26f9a66a946a18f4afe233ed0bbec (127.1.210.66:38745), 190f26f416b74325b4abedc2cffb5422 (127.1.210.67:38123)
W20260806 02:50:15.620428 18727 leader_election.cc:341] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eca26f9a66a946a18f4afe233ed0bbec (127.1.210.66:38745): Not found: Tablet not found: 53f3239c20a943658c00cee68fc56f68
W20260806 02:50:15.620627 18727 leader_election.cc:341] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 190f26f416b74325b4abedc2cffb5422 (127.1.210.67:38123): Not found: Tablet not found: 53f3239c20a943658c00cee68fc56f68
I20260806 02:50:15.620718 18727 leader_election.cc:302] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cffa79db50164fc6a5319c74034efccf; no voters: 190f26f416b74325b4abedc2cffb5422, eca26f9a66a946a18f4afe233ed0bbec
I20260806 02:50:15.620939 19131 raft_consensus.cc:2726] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:50:16.003477 19132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:16.002578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.66:44335)
0806 02:50:16.003180 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:16.003283 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":317,"threads_started":1}
I20260806 02:50:16.477093 19133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:16.476224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.65:54195)
0806 02:50:16.476779 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:16.476868 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":290,"threads_started":1}
I20260806 02:50:16.511082 19134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:16.510181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.67:44435)
0806 02:50:16.510769 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:16.510852 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":335,"threads_started":1}
I20260806 02:50:17.005697 19135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:17.004730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.66:45071)
0806 02:50:17.005386 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:17.005495 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":338,"threads_started":1}
I20260806 02:50:17.479451 19136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:17.478413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.65:60733)
0806 02:50:17.479152 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:17.479263 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":370,"threads_started":1}
I20260806 02:50:17.513374 19137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:17.512236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.67:56051)
0806 02:50:17.512936 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:17.513078 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":311,"threads_started":1}
I20260806 02:50:17.618888 19138 raft_consensus.cc:482] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:50:17.619153 19138 raft_consensus.cc:504] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cffa79db50164fc6a5319c74034efccf" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 41307 } } peers { permanent_uuid: "eca26f9a66a946a18f4afe233ed0bbec" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 38745 } } peers { permanent_uuid: "190f26f416b74325b4abedc2cffb5422" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38123 } }
I20260806 02:50:17.619843 19138 leader_election.cc:288] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eca26f9a66a946a18f4afe233ed0bbec (127.1.210.66:38745), 190f26f416b74325b4abedc2cffb5422 (127.1.210.67:38123)
W20260806 02:50:17.621016 18727 leader_election.cc:341] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 190f26f416b74325b4abedc2cffb5422 (127.1.210.67:38123): Not found: Tablet not found: 53f3239c20a943658c00cee68fc56f68
W20260806 02:50:17.621250 18727 leader_election.cc:341] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eca26f9a66a946a18f4afe233ed0bbec (127.1.210.66:38745): Not found: Tablet not found: 53f3239c20a943658c00cee68fc56f68
I20260806 02:50:17.621336 18727 leader_election.cc:302] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cffa79db50164fc6a5319c74034efccf; no voters: 190f26f416b74325b4abedc2cffb5422, eca26f9a66a946a18f4afe233ed0bbec
I20260806 02:50:17.621596 19138 raft_consensus.cc:2726] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:50:18.007766 19139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.006912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.66:42401)
0806 02:50:18.007474 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:18.007567 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":255,"threads_started":1}
I20260806 02:50:18.481606 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.480758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.65:47351)
0806 02:50:18.481313 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:18.481414 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":290,"threads_started":1}
I20260806 02:50:18.515507 19141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.514615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.67:48277)
0806 02:50:18.515218 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:18.515306 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":358,"threads_started":1}
I20260806 02:50:19.010273 19142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:19.009059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.66:49753)
0806 02:50:19.009912 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:19.010049 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":450,"threads_started":1}
I20260806 02:50:19.483764 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:19.482905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.65:52441)
0806 02:50:19.483480 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:19.483576 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":314,"threads_started":1}
I20260806 02:50:19.517433 19144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:19.516592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.67:33709)
0806 02:50:19.517162 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:19.517262 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":304,"threads_started":1}
I20260806 02:50:20.012706 19145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:20.011723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.66:53635)
0806 02:50:20.012374 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:20.012482 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":384,"threads_started":1}
I20260806 02:50:20.485708 19146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:20.484885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.65:60611)
0806 02:50:20.485412 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:20.485507 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":267,"threads_started":1}
I20260806 02:50:20.519482 19147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:20.518545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.67:39157)
0806 02:50:20.519196 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:20.519288 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":385,"threads_started":1}
I20260806 02:50:21.014909 19148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:21.013957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.66:60007)
0806 02:50:21.014521 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:21.014635 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":297,"threads_started":1}
I20260806 02:50:21.182544 19149 raft_consensus.cc:482] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:50:21.182835 19149 raft_consensus.cc:504] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cffa79db50164fc6a5319c74034efccf" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 41307 } } peers { permanent_uuid: "eca26f9a66a946a18f4afe233ed0bbec" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 38745 } } peers { permanent_uuid: "190f26f416b74325b4abedc2cffb5422" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38123 } }
I20260806 02:50:21.183545 19149 leader_election.cc:288] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eca26f9a66a946a18f4afe233ed0bbec (127.1.210.66:38745), 190f26f416b74325b4abedc2cffb5422 (127.1.210.67:38123)
W20260806 02:50:21.184652 18727 leader_election.cc:341] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eca26f9a66a946a18f4afe233ed0bbec (127.1.210.66:38745): Not found: Tablet not found: 53f3239c20a943658c00cee68fc56f68
W20260806 02:50:21.184856 18727 leader_election.cc:341] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 190f26f416b74325b4abedc2cffb5422 (127.1.210.67:38123): Not found: Tablet not found: 53f3239c20a943658c00cee68fc56f68
I20260806 02:50:21.184950 18727 leader_election.cc:302] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cffa79db50164fc6a5319c74034efccf; no voters: 190f26f416b74325b4abedc2cffb5422, eca26f9a66a946a18f4afe233ed0bbec
I20260806 02:50:21.185210 19149 raft_consensus.cc:2726] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:50:21.488042 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:21.487048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.65:49141)
0806 02:50:21.487600 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:21.487735 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":275,"threads_started":1}
I20260806 02:50:21.521536 19151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:21.520689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.67:35749)
0806 02:50:21.521243 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:21.521362 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":261,"threads_started":1}
I20260806 02:50:22.017058 19152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:22.016112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.66:53779)
0806 02:50:22.016734 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:22.016865 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":293,"threads_started":1}
I20260806 02:50:22.490265 19153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:22.489294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.65:48149)
0806 02:50:22.489829 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:22.490065 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":323,"threads_started":1}
I20260806 02:50:22.523556 19154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:22.522717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.67:42003)
0806 02:50:22.523285 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:22.523378 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":297,"threads_started":1}
I20260806 02:50:23.019173 19155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:23.018266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.66:36195)
0806 02:50:23.018868 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:23.018990 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":287,"threads_started":1}
I20260806 02:50:23.225232 19156 raft_consensus.cc:482] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:50:23.225476 19156 raft_consensus.cc:504] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cffa79db50164fc6a5319c74034efccf" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 41307 } } peers { permanent_uuid: "eca26f9a66a946a18f4afe233ed0bbec" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 38745 } } peers { permanent_uuid: "190f26f416b74325b4abedc2cffb5422" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38123 } }
I20260806 02:50:23.226192 19156 leader_election.cc:288] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eca26f9a66a946a18f4afe233ed0bbec (127.1.210.66:38745), 190f26f416b74325b4abedc2cffb5422 (127.1.210.67:38123)
W20260806 02:50:23.227348 18727 leader_election.cc:341] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eca26f9a66a946a18f4afe233ed0bbec (127.1.210.66:38745): Not found: Tablet not found: 53f3239c20a943658c00cee68fc56f68
W20260806 02:50:23.227588 18727 leader_election.cc:341] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 190f26f416b74325b4abedc2cffb5422 (127.1.210.67:38123): Not found: Tablet not found: 53f3239c20a943658c00cee68fc56f68
I20260806 02:50:23.227725 18727 leader_election.cc:302] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cffa79db50164fc6a5319c74034efccf; no voters: 190f26f416b74325b4abedc2cffb5422, eca26f9a66a946a18f4afe233ed0bbec
I20260806 02:50:23.228029 19156 raft_consensus.cc:2726] T 53f3239c20a943658c00cee68fc56f68 P cffa79db50164fc6a5319c74034efccf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:50:23.492192 19157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:23.491365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.65:52929)
0806 02:50:23.491874 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:23.491971 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":270,"threads_started":1}
I20260806 02:50:23.525501 19158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:23.524684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.67:36639)
0806 02:50:23.525233 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:23.525331 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":306,"threads_started":1}
I20260806 02:50:24.021278 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:24.020267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.66:53295)
0806 02:50:24.020868 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:24.021055 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":321,"threads_started":1}
I20260806 02:50:24.494305 19160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:24.493525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.65:48225)
0806 02:50:24.494007 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:24.494116 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":273,"threads_started":1}
I20260806 02:50:24.527668 19161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:24.526647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.67:37921)
0806 02:50:24.527343 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:24.527439 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":304,"threads_started":1}
I20260806 02:50:25.023524 19162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:25.022635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.66:58103)
0806 02:50:25.023236 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:25.023346 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":256,"threads_started":1}
I20260806 02:50:25.496070 19163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:25.495306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.65:59113)
0806 02:50:25.495788 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:25.495870 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":267,"threads_started":1}
I20260806 02:50:25.505174 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 18666
I20260806 02:50:25.528124 19164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:25.506475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44492 (local address 127.1.210.65:41307)
0806 02:50:25.507161 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:25.507168 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:25.507196 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:50:25.507266 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:25.507273 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:25.507576 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:25.507874 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:25.509776 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:25.510944 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:25.512229 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:25.512822 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:25.512930 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:25.521387 (+ 8457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:25.521409 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:25.521426 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:25.521474 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:25.524852 (+ 3378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:25.525789 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:25.525797 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:25.525799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:25.525949 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:25.526316 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:25.526319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:25.526321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:25.526824 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:25.527075 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:25.527446 (+ 371us) server_negotiation.cc:299] Negotiation successful
0806 02:50:25.527693 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":305,"threads_started":1}
I20260806 02:50:25.529599 18833 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:50:25.529950 19165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:25.529054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:32935 (local address 127.1.210.67:54863)
0806 02:50:25.529609 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:25.529789 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.126:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":313,"threads_started":1}
I20260806 02:50:25.624400 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 18666
I20260806 02:50:25.647640 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 18842
I20260806 02:50:25.662334 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:25.648509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42516 (local address 127.1.210.66:38745)
0806 02:50:25.649053 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:25.649058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:25.649077 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:50:25.649135 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:25.649138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:25.649209 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:25.649311 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:25.650798 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:25.651687 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:25.652990 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:25.653462 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:25.653568 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:25.656419 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:25.656455 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:25.656458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:25.656498 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:25.659546 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:25.660286 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:25.660291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:25.660293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:25.660454 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:25.660905 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:25.660910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:25.660913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:25.661237 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:25.661466 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:25.661802 (+ 336us) server_negotiation.cc:299] Negotiation successful
0806 02:50:25.662037 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":296,"threads_started":1}
I20260806 02:50:25.663344 18966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:50:25.755023 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 18842
I20260806 02:50:25.776587 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 18975
I20260806 02:50:25.791488 19169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:25.777656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40486 (local address 127.1.210.67:38123)
0806 02:50:25.778259 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:25.778263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:25.778281 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:50:25.778345 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:25.778349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:25.778433 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:25.778549 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:25.780042 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:25.780853 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:25.782086 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:25.782552 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:25.782656 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:25.785527 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:25.785533 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:25.785537 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:25.785576 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:25.788709 (+ 3133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:25.789504 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:25.789509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:25.789512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:25.789670 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:25.790046 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:25.790051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:25.790053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:25.790374 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:25.790619 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:25.790929 (+ 310us) server_negotiation.cc:299] Negotiation successful
0806 02:50:25.791176 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":326,"threads_started":1}
I20260806 02:50:25.792187 19101 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:50:25.878496 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 18975
2026-08-06T02:50:25Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65474 ms)
[----------] 30 tests from SecurityITest (321679 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:50:25 dist-test-slave-8tdl krb5kdc[19173](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:50:25 dist-test-slave-8tdl krb5kdc[19173](info): set up 2 sockets
Aug 06 02:50:25 dist-test-slave-8tdl krb5kdc[19173](info): commencing operation
krb5kdc: starting...
W20260806 02:50:27.046569 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.002s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:50:27 dist-test-slave-8tdl krb5kdc[19173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:50:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
I20260806 02:50:27.363286 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:46067
--webserver_interface=127.1.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:39427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:46067
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:27.727946 19186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:27.728461 19186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:27.728636 19186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:27.737327 19186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:27.737476 19186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:27.737584 19186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:27.737684 19186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:27.737748 19186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:27.748484 19186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:39427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:46067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:46067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:50:27.749949 19186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:27.751556 19186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:27.764928 19191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:27.764858 19194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:27.764900 19192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:27.765897 19186 server_base.cc:1034] running on GCE node
I20260806 02:50:27.766912 19186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:27.768546 19186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:27.769793 19186 hybrid_clock.cc:648] HybridClock initialized: now 1785984627769746 us; error 81 us; skew 500 ppm
I20260806 02:50:27.770182 19186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:50:27 dist-test-slave-8tdl krb5kdc[19173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984627, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:27.773734 19186 init.cc:377] Logged in from keytab as kudu/127.1.210.126@KRBTEST.COM (short username kudu)
I20260806 02:50:27.775554 19186 webserver.cc:460] Webserver started at http://127.1.210.126:33791/ using document root <none> and password file <none>
I20260806 02:50:27.776346 19186 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:27.776510 19186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:27.776863 19186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:27.780165 19186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "6ad9f180a0f34a20b4c5574c3212ca97"
format_stamp: "Formatted at 2026-08-06 02:50:27 on dist-test-slave-8tdl"
server_key: "94cad36c1a485fbcf8dcdf409c4edd8e"
server_key_iv: "589ad06b5f7d4447dcba7a56953a2f8d"
server_key_version: "encryptionkey@0"
I20260806 02:50:27.781009 19186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "6ad9f180a0f34a20b4c5574c3212ca97"
format_stamp: "Formatted at 2026-08-06 02:50:27 on dist-test-slave-8tdl"
server_key: "94cad36c1a485fbcf8dcdf409c4edd8e"
server_key_iv: "589ad06b5f7d4447dcba7a56953a2f8d"
server_key_version: "encryptionkey@0"
I20260806 02:50:27.787240 19186 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 02:50:27.791416 19201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:27.793058 19186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:50:27.793284 19186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "6ad9f180a0f34a20b4c5574c3212ca97"
format_stamp: "Formatted at 2026-08-06 02:50:27 on dist-test-slave-8tdl"
server_key: "94cad36c1a485fbcf8dcdf409c4edd8e"
server_key_iv: "589ad06b5f7d4447dcba7a56953a2f8d"
server_key_version: "encryptionkey@0"
I20260806 02:50:27.793542 19186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:27.810144 19186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:27.815088 19186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:27.815412 19186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:27.835896 19186 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:46067
I20260806 02:50:27.835916 19253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:46067 every 8 connection(s)
I20260806 02:50:27.837407 19186 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:27.841897 19254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:27.844794 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 19186
I20260806 02:50:27.845031 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:50:27.845849 1865 external_mini_cluster.cc:1428] Setting key bee0f94630627596d2f6f56ab664f7a4
I20260806 02:50:27.855858 19254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ad9f180a0f34a20b4c5574c3212ca97: Bootstrap starting.
I20260806 02:50:27.860167 19254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ad9f180a0f34a20b4c5574c3212ca97: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:27.861696 19254 log.cc:824] T 00000000000000000000000000000000 P 6ad9f180a0f34a20b4c5574c3212ca97: Log is configured to *not* fsync() on all Append() calls
Aug 06 02:50:27 dist-test-slave-8tdl krb5kdc[19173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:50:27.865723 19254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ad9f180a0f34a20b4c5574c3212ca97: No bootstrap required, opened a new log
I20260806 02:50:27.872742 19257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:27.848299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55996 (local address 127.1.210.126:46067)
0806 02:50:27.849217 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:27.849231 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:27.849274 (+ 43us) server_negotiation.cc:407] Connection header received
0806 02:50:27.851159 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:27.851184 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:27.851646 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:27.852034 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:27.853970 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:27.855445 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:27.857564 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:27.858430 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:27.858587 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:27.864402 (+ 5815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:27.864412 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:27.864431 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:27.864471 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:27.867868 (+ 3397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:27.868908 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:27.868916 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:27.868923 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:27.869128 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:27.869600 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:27.869604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:27.869605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:27.870226 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:27.870478 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:27.871044 (+ 566us) server_negotiation.cc:299] Negotiation successful
0806 02:50:27.871486 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":460,"threads_started":1}
I20260806 02:50:27.876578 19254 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ad9f180a0f34a20b4c5574c3212ca97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad9f180a0f34a20b4c5574c3212ca97" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 46067 } }
I20260806 02:50:27.877044 19254 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ad9f180a0f34a20b4c5574c3212ca97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:27.877170 19254 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ad9f180a0f34a20b4c5574c3212ca97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ad9f180a0f34a20b4c5574c3212ca97, State: Initialized, Role: FOLLOWER
I20260806 02:50:27.877794 19254 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ad9f180a0f34a20b4c5574c3212ca97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad9f180a0f34a20b4c5574c3212ca97" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 46067 } }
I20260806 02:50:27.878080 19254 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ad9f180a0f34a20b4c5574c3212ca97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:50:27.878255 19254 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ad9f180a0f34a20b4c5574c3212ca97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:50:27.878441 19254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ad9f180a0f34a20b4c5574c3212ca97 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:27.881096 19254 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ad9f180a0f34a20b4c5574c3212ca97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad9f180a0f34a20b4c5574c3212ca97" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 46067 } }
I20260806 02:50:27.881711 19254 leader_election.cc:302] T 00000000000000000000000000000000 P 6ad9f180a0f34a20b4c5574c3212ca97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ad9f180a0f34a20b4c5574c3212ca97; no voters:
I20260806 02:50:27.882395 19254 leader_election.cc:288] T 00000000000000000000000000000000 P 6ad9f180a0f34a20b4c5574c3212ca97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:50:27.882540 19260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ad9f180a0f34a20b4c5574c3212ca97 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:27.883440 19260 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ad9f180a0f34a20b4c5574c3212ca97 [term 1 LEADER]: Becoming Leader. State: Replica: 6ad9f180a0f34a20b4c5574c3212ca97, State: Running, Role: LEADER
I20260806 02:50:27.884070 19260 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ad9f180a0f34a20b4c5574c3212ca97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad9f180a0f34a20b4c5574c3212ca97" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 46067 } }
I20260806 02:50:27.884929 19254 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ad9f180a0f34a20b4c5574c3212ca97 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:27.887329 19262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ad9f180a0f34a20b4c5574c3212ca97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ad9f180a0f34a20b4c5574c3212ca97. Latest consensus state: current_term: 1 leader_uuid: "6ad9f180a0f34a20b4c5574c3212ca97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad9f180a0f34a20b4c5574c3212ca97" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 46067 } } }
I20260806 02:50:27.887705 19262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ad9f180a0f34a20b4c5574c3212ca97 [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:27.888675 19261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ad9f180a0f34a20b4c5574c3212ca97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ad9f180a0f34a20b4c5574c3212ca97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad9f180a0f34a20b4c5574c3212ca97" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 46067 } } }
I20260806 02:50:27.889096 19261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ad9f180a0f34a20b4c5574c3212ca97 [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:27.892038 19266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:50:27.900417 19266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:27.909454 19266 catalog_manager.cc:1347] Generated new cluster ID: f94482d542b84225817531cb35716f23
I20260806 02:50:27.909590 19266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:27.925582 19266 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:50:27.927260 19266 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:27.938534 19266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ad9f180a0f34a20b4c5574c3212ca97: Generated new TSK 0
I20260806 02:50:27.939779 19266 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab.
Entry for principal kudu/127.1.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab.
I20260806 02:50:28.023324 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.65:0
--local_ip_for_outbound_sockets=127.1.210.65
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:46067
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:39427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:28.381210 19281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:28.381506 19281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:28.381660 19281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:28.390010 19281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:28.390136 19281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:28.390344 19281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.65
I20260806 02:50:28.401371 19281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:39427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:46067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.65
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:50:28.402882 19281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:28.404354 19281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:28.418185 19287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:28.418262 19286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:28.418239 19289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:28.419683 19281 server_base.cc:1034] running on GCE node
I20260806 02:50:28.420320 19281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:28.421470 19281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:28.422744 19281 hybrid_clock.cc:648] HybridClock initialized: now 1785984628422643 us; error 39 us; skew 500 ppm
I20260806 02:50:28.423092 19281 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:50:28 dist-test-slave-8tdl krb5kdc[19173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984628, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:28.426404 19281 init.cc:377] Logged in from keytab as kudu/127.1.210.65@KRBTEST.COM (short username kudu)
I20260806 02:50:28.428251 19281 webserver.cc:460] Webserver started at http://127.1.210.65:38257/ using document root <none> and password file <none>
I20260806 02:50:28.428996 19281 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:28.429157 19281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:28.429493 19281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:28.432633 19281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-0/data/instance:
uuid: "29e53c8ae6e44cc99ea148beecea4809"
format_stamp: "Formatted at 2026-08-06 02:50:28 on dist-test-slave-8tdl"
server_key: "0a2c8462a49164ba577fd9a1e521943b"
server_key_iv: "fa73eab23db9795f2cac8ccba4de3b5d"
server_key_version: "encryptionkey@0"
I20260806 02:50:28.433471 19281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance:
uuid: "29e53c8ae6e44cc99ea148beecea4809"
format_stamp: "Formatted at 2026-08-06 02:50:28 on dist-test-slave-8tdl"
server_key: "0a2c8462a49164ba577fd9a1e521943b"
server_key_iv: "fa73eab23db9795f2cac8ccba4de3b5d"
server_key_version: "encryptionkey@0"
I20260806 02:50:28.439407 19281 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260806 02:50:28.443570 19296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:28.445083 19281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:50:28.445272 19281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-0/wal
uuid: "29e53c8ae6e44cc99ea148beecea4809"
format_stamp: "Formatted at 2026-08-06 02:50:28 on dist-test-slave-8tdl"
server_key: "0a2c8462a49164ba577fd9a1e521943b"
server_key_iv: "fa73eab23db9795f2cac8ccba4de3b5d"
server_key_version: "encryptionkey@0"
I20260806 02:50:28.445497 19281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:28.464020 19281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:28.468542 19281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:28.468868 19281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:28.469908 19281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:28.471439 19281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:28.471545 19281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:28.471695 19281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:28.471766 19281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:28.507962 19281 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.65:35991
I20260806 02:50:28.508018 19409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.65:35991 every 8 connection(s)
I20260806 02:50:28.509465 19281 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
I20260806 02:50:28.516836 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 19281
I20260806 02:50:28.517071 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance
I20260806 02:50:28.517949 1865 external_mini_cluster.cc:1428] Setting key 2006ae488ebb4e907d55f38bcf0bbe11
Aug 06 02:50:28 dist-test-slave-8tdl krb5kdc[19173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984628, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:50:28.533571 19412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:28.512141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:43293 (local address 127.1.210.126:46067)
0806 02:50:28.512655 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:28.512660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:28.513475 (+ 815us) server_negotiation.cc:407] Connection header received
0806 02:50:28.515535 (+ 2060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:28.515541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:28.515636 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:28.515754 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:28.517916 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:28.518900 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:28.521216 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:28.521654 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:28.521763 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:28.526817 (+ 5054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:28.526822 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:28.526826 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:28.526863 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:28.529696 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:28.530768 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:28.530772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:28.530774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:28.530899 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:28.531412 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:28.531415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:28.531417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:28.531700 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:28.531814 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:28.532854 (+ 1040us) server_negotiation.cc:299] Negotiation successful
0806 02:50:28.533125 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":296,"threads_started":1}
I20260806 02:50:28.534310 19413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:28.512557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:46067 (local address 127.1.210.65:43293)
0806 02:50:28.513302 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:28.513342 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:28.515251 (+ 1909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:28.516013 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:28.516021 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:28.516766 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:28.517670 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:28.517700 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:28.519080 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:28.519086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:28.521033 (+ 1947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:28.521055 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:28.521921 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:28.521931 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:28.522336 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:28.523244 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:28.523288 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:28.526482 (+ 3194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:28.529867 (+ 3385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:28.529879 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:28.529901 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:28.530608 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:28.531012 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:28.531019 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:28.531025 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:28.531300 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:28.531829 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:28.531849 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:28.532396 (+ 547us) client_negotiation.cc:769] Sending connection context
0806 02:50:28.532678 (+ 282us) client_negotiation.cc:240] Negotiation successful
0806 02:50:28.533140 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":302,"threads_started":1}
I20260806 02:50:28.536020 19410 heartbeater.cc:347] Connected to a master server at 127.1.210.126:46067
I20260806 02:50:28.536342 19410 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:28.537106 19410 heartbeater.cc:511] Master 127.1.210.126:46067 requested a full tablet report, sending...
I20260806 02:50:28.539417 19219 ts_manager.cc:194] Registered new tserver with Master: 29e53c8ae6e44cc99ea148beecea4809 (127.1.210.65:35991)
I20260806 02:50:28.540911 19219 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.65@KRBTEST.COM'} at 127.1.210.65:43293
WARNING: no policy specified for kudu/127.1.210.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab.
Entry for principal kudu/127.1.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab.
I20260806 02:50:28.586952 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.66:0
--local_ip_for_outbound_sockets=127.1.210.66
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:46067
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:39427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:28.960696 19416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:28.961156 19416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:28.961273 19416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:28.968923 19416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:28.969049 19416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:28.969167 19416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.66
I20260806 02:50:28.979367 19416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:39427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:46067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.66
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:50:28.980686 19416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:28.982079 19416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:28.996562 19421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:28.996523 19424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:28.996598 19422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:28.997499 19416 server_base.cc:1034] running on GCE node
I20260806 02:50:28.998234 19416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:28.999328 19416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:29.000562 19416 hybrid_clock.cc:648] HybridClock initialized: now 1785984629000505 us; error 43 us; skew 500 ppm
I20260806 02:50:29.000895 19416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:50:28 dist-test-slave-8tdl krb5kdc[19173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984628, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:29.004748 19416 init.cc:377] Logged in from keytab as kudu/127.1.210.66@KRBTEST.COM (short username kudu)
I20260806 02:50:29.006630 19416 webserver.cc:460] Webserver started at http://127.1.210.66:33057/ using document root <none> and password file <none>
I20260806 02:50:29.007480 19416 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:29.007622 19416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:29.007972 19416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:29.011340 19416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-1/data/instance:
uuid: "25e3e6227d894624831fb473dda136e6"
format_stamp: "Formatted at 2026-08-06 02:50:29 on dist-test-slave-8tdl"
server_key: "2169776aa81763ebb8a4bebee3dc2ec4"
server_key_iv: "bda385d3d3647b3d580e006b3b7ec18f"
server_key_version: "encryptionkey@0"
I20260806 02:50:29.012254 19416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance:
uuid: "25e3e6227d894624831fb473dda136e6"
format_stamp: "Formatted at 2026-08-06 02:50:29 on dist-test-slave-8tdl"
server_key: "2169776aa81763ebb8a4bebee3dc2ec4"
server_key_iv: "bda385d3d3647b3d580e006b3b7ec18f"
server_key_version: "encryptionkey@0"
I20260806 02:50:29.018318 19416 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260806 02:50:29.022317 19431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:29.024115 19416 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:50:29.024304 19416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-1/wal
uuid: "25e3e6227d894624831fb473dda136e6"
format_stamp: "Formatted at 2026-08-06 02:50:29 on dist-test-slave-8tdl"
server_key: "2169776aa81763ebb8a4bebee3dc2ec4"
server_key_iv: "bda385d3d3647b3d580e006b3b7ec18f"
server_key_version: "encryptionkey@0"
I20260806 02:50:29.024520 19416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:29.045534 19416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:29.050500 19416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:29.050941 19416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:29.052033 19416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:29.053504 19416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:29.053582 19416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:29.053742 19416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:29.053795 19416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:29.089919 19416 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.66:41025
I20260806 02:50:29.089972 19545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.66:41025 every 8 connection(s)
I20260806 02:50:29.091361 19416 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
I20260806 02:50:29.099531 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 19416
I20260806 02:50:29.099742 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance
I20260806 02:50:29.100303 1865 external_mini_cluster.cc:1428] Setting key 0b435d40823d49c1928e9494c9f604ee
Aug 06 02:50:29 dist-test-slave-8tdl krb5kdc[19173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984628, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:50:29.113830 19548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:29.094158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:38609 (local address 127.1.210.126:46067)
0806 02:50:29.094638 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:29.094642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:29.095866 (+ 1224us) server_negotiation.cc:407] Connection header received
0806 02:50:29.097891 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:29.097897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:29.098037 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:29.098177 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:29.100125 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.101051 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.102937 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.103389 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.103530 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:29.107350 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:29.107355 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:29.107359 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:29.107393 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:29.110127 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:29.111038 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:29.111043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:29.111045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:29.111231 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:29.111895 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:29.111900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:29.111904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:29.112214 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:29.112383 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:29.113170 (+ 787us) server_negotiation.cc:299] Negotiation successful
0806 02:50:29.113438 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":257,"threads_started":1}
I20260806 02:50:29.114990 19549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:29.094839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:46067 (local address 127.1.210.66:38609)
0806 02:50:29.095698 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:29.095745 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:29.097611 (+ 1866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:29.098419 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:29.098428 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:29.099027 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:29.099905 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:29.099935 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.101277 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.101281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:29.102790 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:29.102804 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.103638 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.103646 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:29.103960 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:29.104710 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:29.104740 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:29.107100 (+ 2360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:29.110298 (+ 3198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:29.110309 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:29.110327 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:29.110895 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:29.111501 (+ 606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:29.111508 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:29.111512 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:29.111756 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:29.112355 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:29.112364 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:29.112764 (+ 400us) client_negotiation.cc:769] Sending connection context
0806 02:50:29.113067 (+ 303us) client_negotiation.cc:240] Negotiation successful
0806 02:50:29.113665 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":377,"threads_started":1}
I20260806 02:50:29.116640 19546 heartbeater.cc:347] Connected to a master server at 127.1.210.126:46067
I20260806 02:50:29.117013 19546 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:29.117997 19546 heartbeater.cc:511] Master 127.1.210.126:46067 requested a full tablet report, sending...
I20260806 02:50:29.120014 19219 ts_manager.cc:194] Registered new tserver with Master: 25e3e6227d894624831fb473dda136e6 (127.1.210.66:41025)
I20260806 02:50:29.121392 19219 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.66@KRBTEST.COM'} at 127.1.210.66:38609
WARNING: no policy specified for kudu/127.1.210.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab.
Entry for principal kudu/127.1.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab.
I20260806 02:50:29.171164 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.67:0
--local_ip_for_outbound_sockets=127.1.210.67
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:46067
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:39427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:29.520143 19552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:29.520428 19552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:29.520525 19552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:29.528137 19552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:29.528264 19552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:29.528381 19552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.67
I20260806 02:50:29.538439 19552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:39427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:46067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.67
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:50:29.539764 19552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:29.541345 19552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:50:29.543900 19410 heartbeater.cc:503] Master 127.1.210.126:46067 was elected leader, sending a full tablet report...
W20260806 02:50:29.555780 19557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:29.555846 19560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:29.555689 19558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:29.557354 19552 server_base.cc:1034] running on GCE node
I20260806 02:50:29.558090 19552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:29.559238 19552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:29.560518 19552 hybrid_clock.cc:648] HybridClock initialized: now 1785984629560449 us; error 51 us; skew 500 ppm
I20260806 02:50:29.560889 19552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:50:29 dist-test-slave-8tdl krb5kdc[19173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984629, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:29.564584 19552 init.cc:377] Logged in from keytab as kudu/127.1.210.67@KRBTEST.COM (short username kudu)
I20260806 02:50:29.566229 19552 webserver.cc:460] Webserver started at http://127.1.210.67:36213/ using document root <none> and password file <none>
I20260806 02:50:29.567028 19552 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:29.567176 19552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:29.567497 19552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:29.570793 19552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-2/data/instance:
uuid: "b4a9b92b06f944ada0cae2be3bd2d89f"
format_stamp: "Formatted at 2026-08-06 02:50:29 on dist-test-slave-8tdl"
server_key: "79a59a42e25a4b47fd1a9c242ffde77a"
server_key_iv: "4aa10952177ae57714ff964613f9b045"
server_key_version: "encryptionkey@0"
I20260806 02:50:29.571746 19552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance:
uuid: "b4a9b92b06f944ada0cae2be3bd2d89f"
format_stamp: "Formatted at 2026-08-06 02:50:29 on dist-test-slave-8tdl"
server_key: "79a59a42e25a4b47fd1a9c242ffde77a"
server_key_iv: "4aa10952177ae57714ff964613f9b045"
server_key_version: "encryptionkey@0"
I20260806 02:50:29.577234 19552 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 02:50:29.580919 19567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:29.582340 19552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:50:29.582547 19552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-2/wal
uuid: "b4a9b92b06f944ada0cae2be3bd2d89f"
format_stamp: "Formatted at 2026-08-06 02:50:29 on dist-test-slave-8tdl"
server_key: "79a59a42e25a4b47fd1a9c242ffde77a"
server_key_iv: "4aa10952177ae57714ff964613f9b045"
server_key_version: "encryptionkey@0"
I20260806 02:50:29.582810 19552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:29.597662 19552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:29.602593 19552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:29.602977 19552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:29.604080 19552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:29.605669 19552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:29.605805 19552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:29.605970 19552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:29.606061 19552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:29.641619 19552 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.67:38805
I20260806 02:50:29.641695 19680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.67:38805 every 8 connection(s)
I20260806 02:50:29.643348 19552 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
I20260806 02:50:29.652655 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 19552
I20260806 02:50:29.652868 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance
I20260806 02:50:29.653522 1865 external_mini_cluster.cc:1428] Setting key 538fb068c870616dd730b60e05d7cd50
Aug 06 02:50:29 dist-test-slave-8tdl krb5kdc[19173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984629, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:50:29.664963 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:29.646170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:53053 (local address 127.1.210.126:46067)
0806 02:50:29.646644 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:29.646648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:29.647771 (+ 1123us) server_negotiation.cc:407] Connection header received
0806 02:50:29.649787 (+ 2016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:29.649793 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:29.649897 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:29.650028 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:29.651715 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.652535 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.654627 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.655323 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.655477 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:29.659299 (+ 3822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:29.659304 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:29.659307 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:29.659341 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:29.661996 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:29.662791 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:29.662795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:29.662797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:29.662972 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:29.663381 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:29.663384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:29.663386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:29.663661 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:29.663784 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:29.664479 (+ 695us) server_negotiation.cc:299] Negotiation successful
0806 02:50:29.664660 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":227,"threads_started":1}
I20260806 02:50:29.665944 19684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:29.646722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:46067 (local address 127.1.210.67:53053)
0806 02:50:29.647601 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:29.647647 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:29.649506 (+ 1859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:29.650284 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:29.650293 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:29.650941 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:29.651539 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:29.651557 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.652741 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.652749 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:29.654444 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:29.654458 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.655616 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.655627 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:29.655919 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:29.656677 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:29.656712 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:29.659082 (+ 2370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:29.662141 (+ 3059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:29.662149 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:29.662161 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:29.662613 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:29.663079 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:29.663082 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:29.663085 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:29.663279 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:29.663772 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:29.663778 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:29.664127 (+ 349us) client_negotiation.cc:769] Sending connection context
0806 02:50:29.664356 (+ 229us) client_negotiation.cc:240] Negotiation successful
0806 02:50:29.664789 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":402,"threads_started":1}
I20260806 02:50:29.667641 19681 heartbeater.cc:347] Connected to a master server at 127.1.210.126:46067
I20260806 02:50:29.668016 19681 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:29.668922 19681 heartbeater.cc:511] Master 127.1.210.126:46067 requested a full tablet report, sending...
I20260806 02:50:29.670430 19219 ts_manager.cc:194] Registered new tserver with Master: b4a9b92b06f944ada0cae2be3bd2d89f (127.1.210.67:38805)
I20260806 02:50:29.671361 19219 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.67@KRBTEST.COM'} at 127.1.210.67:53053
I20260806 02:50:29.682000 1865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:50:29.700663 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:29.687162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56002 (local address 127.1.210.126:46067)
0806 02:50:29.687415 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:29.687425 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:29.687996 (+ 571us) server_negotiation.cc:407] Connection header received
0806 02:50:29.688202 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:29.688207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:29.688296 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:29.688406 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:29.690127 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.690951 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.692330 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.692747 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.692838 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:29.695394 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:29.695399 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:29.695402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:29.695435 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:29.698091 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:29.698818 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:29.698822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:29.698824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:29.698989 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:29.699330 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:29.699333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:29.699335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:29.699656 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:29.699833 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:29.700125 (+ 292us) server_negotiation.cc:299] Negotiation successful
0806 02:50:29.700301 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260806 02:50:29.704962 19219 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56002:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:50:29.711928 19219 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:50:29.740489 19695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:29.729191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47598 (local address 127.1.210.66:41025)
0806 02:50:29.729826 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:29.729836 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:29.729867 (+ 31us) server_negotiation.cc:407] Connection header received
0806 02:50:29.730157 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:29.730164 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:29.730370 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:29.731919 (+ 1549us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:29.732230 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.734370 (+ 2140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.737405 (+ 3035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.739196 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.739351 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:29.739588 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:29.739930 (+ 342us) server_negotiation.cc:299] Negotiation successful
0806 02:50:29.740103 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":290,"threads_started":1}
I20260806 02:50:29.740617 19694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:29.728940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:41025 (local address 127.0.0.1:47598)
0806 02:50:29.729687 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:29.729738 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:29.729918 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:29.730625 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:29.730633 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:29.730670 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:29.731195 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:29.731224 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.734554 (+ 3330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.734564 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:29.737099 (+ 2535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:29.737128 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.739405 (+ 2277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.739412 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:29.739689 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:29.739735 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 02:50:29.739882 (+ 147us) client_negotiation.cc:240] Negotiation successful
0806 02:50:29.740103 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":25,"thread_start_us":809,"threads_started":1}
I20260806 02:50:29.743417 19696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:29.730517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:35991 (local address 127.0.0.1:33186)
0806 02:50:29.731182 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:29.731200 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:29.731315 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:29.735758 (+ 4443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:29.735763 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:29.735789 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:29.736327 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:29.736339 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.738226 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.738229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:29.740520 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:29.740535 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.742432 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.742439 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:29.742660 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:29.742727 (+ 67us) client_negotiation.cc:769] Sending connection context
0806 02:50:29.742826 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 02:50:29.742970 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":39,"thread_start_us":350,"threads_started":1}
I20260806 02:50:29.744058 19698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:29.731264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:38805 (local address 127.0.0.1:53866)
0806 02:50:29.731880 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:29.731902 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:29.732036 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:29.736102 (+ 4066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:29.736113 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:29.736141 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:29.736576 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:29.736589 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.738566 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.738576 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:29.741069 (+ 2493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:29.741084 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.743206 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.743212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:29.743398 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:29.743436 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 02:50:29.743533 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 02:50:29.743679 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":2,"thread_start_us":375,"threads_started":1}
I20260806 02:50:29.744098 19699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:29.731857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53866 (local address 127.1.210.67:38805)
0806 02:50:29.735477 (+ 3620us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:29.735485 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:29.735517 (+ 32us) server_negotiation.cc:407] Connection header received
0806 02:50:29.735623 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:29.735630 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:29.735868 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:29.736056 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:29.736759 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.738429 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.741234 (+ 2805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.742661 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.742819 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:29.743042 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:29.743603 (+ 561us) server_negotiation.cc:299] Negotiation successful
0806 02:50:29.743772 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3436,"thread_start_us":3278,"threads_started":1}
I20260806 02:50:29.744217 19697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:29.730941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33186 (local address 127.1.210.65:35991)
0806 02:50:29.735194 (+ 4253us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:29.735204 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:29.735235 (+ 31us) server_negotiation.cc:407] Connection header received
0806 02:50:29.735333 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:29.735343 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:29.735573 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:29.735779 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:29.736595 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.738061 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.740737 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.741903 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.742052 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:29.742295 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:29.743719 (+ 1424us) server_negotiation.cc:299] Negotiation successful
0806 02:50:29.743901 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4022,"thread_start_us":3793,"threads_started":1}
I20260806 02:50:29.745280 19480 tablet_service.cc:1467] Processing CreateTablet for tablet bcb19dc2950241f2af398bbd608947bc (DEFAULT_TABLE table=test-table [id=b8632038629e40a78bc76d61044f4400]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:29.746542 19480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcb19dc2950241f2af398bbd608947bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:29.747849 19344 tablet_service.cc:1467] Processing CreateTablet for tablet bcb19dc2950241f2af398bbd608947bc (DEFAULT_TABLE table=test-table [id=b8632038629e40a78bc76d61044f4400]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:29.749727 19344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcb19dc2950241f2af398bbd608947bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:29.747848 19615 tablet_service.cc:1467] Processing CreateTablet for tablet bcb19dc2950241f2af398bbd608947bc (DEFAULT_TABLE table=test-table [id=b8632038629e40a78bc76d61044f4400]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:29.751241 19615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcb19dc2950241f2af398bbd608947bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:29.761554 19700 tablet_bootstrap.cc:492] T bcb19dc2950241f2af398bbd608947bc P 29e53c8ae6e44cc99ea148beecea4809: Bootstrap starting.
I20260806 02:50:29.765450 19700 tablet_bootstrap.cc:654] T bcb19dc2950241f2af398bbd608947bc P 29e53c8ae6e44cc99ea148beecea4809: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:29.767616 19700 log.cc:824] T bcb19dc2950241f2af398bbd608947bc P 29e53c8ae6e44cc99ea148beecea4809: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:29.767890 19701 tablet_bootstrap.cc:492] T bcb19dc2950241f2af398bbd608947bc P 25e3e6227d894624831fb473dda136e6: Bootstrap starting.
I20260806 02:50:29.768234 19702 tablet_bootstrap.cc:492] T bcb19dc2950241f2af398bbd608947bc P b4a9b92b06f944ada0cae2be3bd2d89f: Bootstrap starting.
I20260806 02:50:29.771700 19701 tablet_bootstrap.cc:654] T bcb19dc2950241f2af398bbd608947bc P 25e3e6227d894624831fb473dda136e6: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:29.772243 19702 tablet_bootstrap.cc:654] T bcb19dc2950241f2af398bbd608947bc P b4a9b92b06f944ada0cae2be3bd2d89f: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:29.773061 19701 log.cc:824] T bcb19dc2950241f2af398bbd608947bc P 25e3e6227d894624831fb473dda136e6: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:29.773285 19702 log.cc:824] T bcb19dc2950241f2af398bbd608947bc P b4a9b92b06f944ada0cae2be3bd2d89f: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:29.773566 19700 tablet_bootstrap.cc:492] T bcb19dc2950241f2af398bbd608947bc P 29e53c8ae6e44cc99ea148beecea4809: No bootstrap required, opened a new log
I20260806 02:50:29.773912 19700 ts_tablet_manager.cc:1397] T bcb19dc2950241f2af398bbd608947bc P 29e53c8ae6e44cc99ea148beecea4809: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260806 02:50:29.776629 19701 tablet_bootstrap.cc:492] T bcb19dc2950241f2af398bbd608947bc P 25e3e6227d894624831fb473dda136e6: No bootstrap required, opened a new log
I20260806 02:50:29.777004 19701 ts_tablet_manager.cc:1397] T bcb19dc2950241f2af398bbd608947bc P 25e3e6227d894624831fb473dda136e6: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260806 02:50:29.777323 19702 tablet_bootstrap.cc:492] T bcb19dc2950241f2af398bbd608947bc P b4a9b92b06f944ada0cae2be3bd2d89f: No bootstrap required, opened a new log
I20260806 02:50:29.777603 19702 ts_tablet_manager.cc:1397] T bcb19dc2950241f2af398bbd608947bc P b4a9b92b06f944ada0cae2be3bd2d89f: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260806 02:50:29.784884 19702 raft_consensus.cc:348] T bcb19dc2950241f2af398bbd608947bc P b4a9b92b06f944ada0cae2be3bd2d89f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e3e6227d894624831fb473dda136e6" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 41025 } } peers { permanent_uuid: "29e53c8ae6e44cc99ea148beecea4809" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 35991 } } peers { permanent_uuid: "b4a9b92b06f944ada0cae2be3bd2d89f" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38805 } }
I20260806 02:50:29.784886 19700 raft_consensus.cc:348] T bcb19dc2950241f2af398bbd608947bc P 29e53c8ae6e44cc99ea148beecea4809 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e3e6227d894624831fb473dda136e6" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 41025 } } peers { permanent_uuid: "29e53c8ae6e44cc99ea148beecea4809" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 35991 } } peers { permanent_uuid: "b4a9b92b06f944ada0cae2be3bd2d89f" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38805 } }
I20260806 02:50:29.785360 19700 raft_consensus.cc:374] T bcb19dc2950241f2af398bbd608947bc P 29e53c8ae6e44cc99ea148beecea4809 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:29.785360 19702 raft_consensus.cc:374] T bcb19dc2950241f2af398bbd608947bc P b4a9b92b06f944ada0cae2be3bd2d89f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:29.785503 19702 raft_consensus.cc:729] T bcb19dc2950241f2af398bbd608947bc P b4a9b92b06f944ada0cae2be3bd2d89f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4a9b92b06f944ada0cae2be3bd2d89f, State: Initialized, Role: FOLLOWER
I20260806 02:50:29.785511 19700 raft_consensus.cc:729] T bcb19dc2950241f2af398bbd608947bc P 29e53c8ae6e44cc99ea148beecea4809 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29e53c8ae6e44cc99ea148beecea4809, State: Initialized, Role: FOLLOWER
I20260806 02:50:29.786247 19702 consensus_queue.cc:260] T bcb19dc2950241f2af398bbd608947bc P b4a9b92b06f944ada0cae2be3bd2d89f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e3e6227d894624831fb473dda136e6" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 41025 } } peers { permanent_uuid: "29e53c8ae6e44cc99ea148beecea4809" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 35991 } } peers { permanent_uuid: "b4a9b92b06f944ada0cae2be3bd2d89f" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38805 } }
I20260806 02:50:29.786263 19700 consensus_queue.cc:260] T bcb19dc2950241f2af398bbd608947bc P 29e53c8ae6e44cc99ea148beecea4809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e3e6227d894624831fb473dda136e6" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 41025 } } peers { permanent_uuid: "29e53c8ae6e44cc99ea148beecea4809" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 35991 } } peers { permanent_uuid: "b4a9b92b06f944ada0cae2be3bd2d89f" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38805 } }
I20260806 02:50:29.787446 19681 heartbeater.cc:503] Master 127.1.210.126:46067 was elected leader, sending a full tablet report...
I20260806 02:50:29.788197 19702 ts_tablet_manager.cc:1428] T bcb19dc2950241f2af398bbd608947bc P b4a9b92b06f944ada0cae2be3bd2d89f: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260806 02:50:29.788203 19701 raft_consensus.cc:348] T bcb19dc2950241f2af398bbd608947bc P 25e3e6227d894624831fb473dda136e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e3e6227d894624831fb473dda136e6" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 41025 } } peers { permanent_uuid: "29e53c8ae6e44cc99ea148beecea4809" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 35991 } } peers { permanent_uuid: "b4a9b92b06f944ada0cae2be3bd2d89f" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38805 } }
I20260806 02:50:29.788758 19701 raft_consensus.cc:374] T bcb19dc2950241f2af398bbd608947bc P 25e3e6227d894624831fb473dda136e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:29.788865 19701 raft_consensus.cc:729] T bcb19dc2950241f2af398bbd608947bc P 25e3e6227d894624831fb473dda136e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25e3e6227d894624831fb473dda136e6, State: Initialized, Role: FOLLOWER
I20260806 02:50:29.789582 19701 consensus_queue.cc:260] T bcb19dc2950241f2af398bbd608947bc P 25e3e6227d894624831fb473dda136e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e3e6227d894624831fb473dda136e6" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 41025 } } peers { permanent_uuid: "29e53c8ae6e44cc99ea148beecea4809" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 35991 } } peers { permanent_uuid: "b4a9b92b06f944ada0cae2be3bd2d89f" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38805 } }
I20260806 02:50:29.790014 19700 ts_tablet_manager.cc:1428] T bcb19dc2950241f2af398bbd608947bc P 29e53c8ae6e44cc99ea148beecea4809: Time spent starting tablet: real 0.016s user 0.013s sys 0.005s
I20260806 02:50:29.791374 19546 heartbeater.cc:503] Master 127.1.210.126:46067 was elected leader, sending a full tablet report...
I20260806 02:50:29.792634 19701 ts_tablet_manager.cc:1428] T bcb19dc2950241f2af398bbd608947bc P 25e3e6227d894624831fb473dda136e6: Time spent starting tablet: real 0.015s user 0.007s sys 0.007s
W20260806 02:50:29.843504 19547 tablet.cc:2369] T bcb19dc2950241f2af398bbd608947bc P 25e3e6227d894624831fb473dda136e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:50:29.868757 19707 raft_consensus.cc:482] T bcb19dc2950241f2af398bbd608947bc P 29e53c8ae6e44cc99ea148beecea4809 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:50:29.869040 19707 raft_consensus.cc:504] T bcb19dc2950241f2af398bbd608947bc P 29e53c8ae6e44cc99ea148beecea4809 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e3e6227d894624831fb473dda136e6" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 41025 } } peers { permanent_uuid: "29e53c8ae6e44cc99ea148beecea4809" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 35991 } } peers { permanent_uuid: "b4a9b92b06f944ada0cae2be3bd2d89f" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38805 } }
I20260806 02:50:29.870850 19707 leader_election.cc:288] T bcb19dc2950241f2af398bbd608947bc P 29e53c8ae6e44cc99ea148beecea4809 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25e3e6227d894624831fb473dda136e6 (127.1.210.66:41025), b4a9b92b06f944ada0cae2be3bd2d89f (127.1.210.67:38805)
I20260806 02:50:29.878189 19695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:29.871212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:47691 (local address 127.1.210.66:41025)
0806 02:50:29.871462 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:29.871472 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:29.872105 (+ 633us) server_negotiation.cc:407] Connection header received
0806 02:50:29.872337 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:29.872344 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:29.872451 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:29.872553 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:29.873277 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.874253 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.876271 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.877238 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.877428 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:29.877514 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:29.877781 (+ 267us) server_negotiation.cc:299] Negotiation successful
0806 02:50:29.877909 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 02:50:29.878449 19709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:29.871273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:41025 (local address 127.1.210.65:47691)
0806 02:50:29.871972 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:29.871992 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:29.872178 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:29.872677 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:29.872681 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:29.872708 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:29.873091 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:29.873103 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.874392 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.874398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:29.876085 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:29.876098 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.877452 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.877458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:29.877617 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:29.877641 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 02:50:29.877755 (+ 114us) client_negotiation.cc:240] Negotiation successful
0806 02:50:29.877894 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":389,"threads_started":1}
I20260806 02:50:29.878485 19699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:29.872397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:41163 (local address 127.1.210.67:38805)
0806 02:50:29.872600 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:29.872618 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:29.872893 (+ 275us) server_negotiation.cc:407] Connection header received
0806 02:50:29.873089 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:29.873094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:29.873181 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:29.873275 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:29.873862 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.874827 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.876751 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.877532 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.877720 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:29.877799 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:29.878105 (+ 306us) server_negotiation.cc:299] Negotiation successful
0806 02:50:29.878241 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 02:50:29.878520 19710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:29.872087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:38805 (local address 127.1.210.65:41163)
0806 02:50:29.872788 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:29.872805 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:29.872908 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:29.873301 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:29.873305 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:29.873333 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:29.873709 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:29.873722 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.875014 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.875018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:29.876527 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:29.876543 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.877783 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.877789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:29.877941 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:29.877964 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 02:50:29.878069 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 02:50:29.878191 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":330,"threads_started":1}
I20260806 02:50:29.879603 19500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcb19dc2950241f2af398bbd608947bc" candidate_uuid: "29e53c8ae6e44cc99ea148beecea4809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25e3e6227d894624831fb473dda136e6" is_pre_election: true
I20260806 02:50:29.879742 19635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcb19dc2950241f2af398bbd608947bc" candidate_uuid: "29e53c8ae6e44cc99ea148beecea4809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4a9b92b06f944ada0cae2be3bd2d89f" is_pre_election: true
I20260806 02:50:29.880195 19500 raft_consensus.cc:2454] T bcb19dc2950241f2af398bbd608947bc P 25e3e6227d894624831fb473dda136e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29e53c8ae6e44cc99ea148beecea4809 in term 0.
I20260806 02:50:29.880256 19635 raft_consensus.cc:2454] T bcb19dc2950241f2af398bbd608947bc P b4a9b92b06f944ada0cae2be3bd2d89f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29e53c8ae6e44cc99ea148beecea4809 in term 0.
I20260806 02:50:29.880939 19300 leader_election.cc:302] T bcb19dc2950241f2af398bbd608947bc P 29e53c8ae6e44cc99ea148beecea4809 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25e3e6227d894624831fb473dda136e6, 29e53c8ae6e44cc99ea148beecea4809; no voters:
I20260806 02:50:29.881356 19707 raft_consensus.cc:2781] T bcb19dc2950241f2af398bbd608947bc P 29e53c8ae6e44cc99ea148beecea4809 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:50:29.881536 19707 raft_consensus.cc:482] T bcb19dc2950241f2af398bbd608947bc P 29e53c8ae6e44cc99ea148beecea4809 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:50:29.881664 19707 raft_consensus.cc:3037] T bcb19dc2950241f2af398bbd608947bc P 29e53c8ae6e44cc99ea148beecea4809 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:29.884146 19707 raft_consensus.cc:504] T bcb19dc2950241f2af398bbd608947bc P 29e53c8ae6e44cc99ea148beecea4809 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e3e6227d894624831fb473dda136e6" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 41025 } } peers { permanent_uuid: "29e53c8ae6e44cc99ea148beecea4809" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 35991 } } peers { permanent_uuid: "b4a9b92b06f944ada0cae2be3bd2d89f" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38805 } }
I20260806 02:50:29.884876 19707 leader_election.cc:288] T bcb19dc2950241f2af398bbd608947bc P 29e53c8ae6e44cc99ea148beecea4809 [CANDIDATE]: Term 1 election: Requested vote from peers 25e3e6227d894624831fb473dda136e6 (127.1.210.66:41025), b4a9b92b06f944ada0cae2be3bd2d89f (127.1.210.67:38805)
I20260806 02:50:29.885375 19500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcb19dc2950241f2af398bbd608947bc" candidate_uuid: "29e53c8ae6e44cc99ea148beecea4809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25e3e6227d894624831fb473dda136e6"
I20260806 02:50:29.885521 19635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcb19dc2950241f2af398bbd608947bc" candidate_uuid: "29e53c8ae6e44cc99ea148beecea4809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4a9b92b06f944ada0cae2be3bd2d89f"
I20260806 02:50:29.885624 19500 raft_consensus.cc:3037] T bcb19dc2950241f2af398bbd608947bc P 25e3e6227d894624831fb473dda136e6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:29.885751 19635 raft_consensus.cc:3037] T bcb19dc2950241f2af398bbd608947bc P b4a9b92b06f944ada0cae2be3bd2d89f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:29.888265 19500 raft_consensus.cc:2454] T bcb19dc2950241f2af398bbd608947bc P 25e3e6227d894624831fb473dda136e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29e53c8ae6e44cc99ea148beecea4809 in term 1.
I20260806 02:50:29.888444 19635 raft_consensus.cc:2454] T bcb19dc2950241f2af398bbd608947bc P b4a9b92b06f944ada0cae2be3bd2d89f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29e53c8ae6e44cc99ea148beecea4809 in term 1.
I20260806 02:50:29.888713 19300 leader_election.cc:302] T bcb19dc2950241f2af398bbd608947bc P 29e53c8ae6e44cc99ea148beecea4809 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25e3e6227d894624831fb473dda136e6, 29e53c8ae6e44cc99ea148beecea4809; no voters:
I20260806 02:50:29.889002 19707 raft_consensus.cc:2781] T bcb19dc2950241f2af398bbd608947bc P 29e53c8ae6e44cc99ea148beecea4809 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:29.889520 19707 raft_consensus.cc:686] T bcb19dc2950241f2af398bbd608947bc P 29e53c8ae6e44cc99ea148beecea4809 [term 1 LEADER]: Becoming Leader. State: Replica: 29e53c8ae6e44cc99ea148beecea4809, State: Running, Role: LEADER
I20260806 02:50:29.890107 19707 consensus_queue.cc:237] T bcb19dc2950241f2af398bbd608947bc P 29e53c8ae6e44cc99ea148beecea4809 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e3e6227d894624831fb473dda136e6" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 41025 } } peers { permanent_uuid: "29e53c8ae6e44cc99ea148beecea4809" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 35991 } } peers { permanent_uuid: "b4a9b92b06f944ada0cae2be3bd2d89f" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38805 } }
W20260806 02:50:29.894984 19682 tablet.cc:2369] T bcb19dc2950241f2af398bbd608947bc P b4a9b92b06f944ada0cae2be3bd2d89f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:50:29.895637 19219 catalog_manager.cc:5495] T bcb19dc2950241f2af398bbd608947bc P 29e53c8ae6e44cc99ea148beecea4809 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29e53c8ae6e44cc99ea148beecea4809 (127.1.210.65). New cstate: current_term: 1 leader_uuid: "29e53c8ae6e44cc99ea148beecea4809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e3e6227d894624831fb473dda136e6" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 41025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29e53c8ae6e44cc99ea148beecea4809" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 35991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4a9b92b06f944ada0cae2be3bd2d89f" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38805 } health_report { overall_health: UNKNOWN } } }
I20260806 02:50:29.933215 19697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:29.927523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33190 (local address 127.1.210.65:35991)
0806 02:50:29.927714 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:29.927718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:29.927809 (+ 91us) server_negotiation.cc:407] Connection header received
0806 02:50:29.928021 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:29.928024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:29.928131 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:29.928271 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:50:29.928730 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.929465 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.931074 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:29.931518 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:29.931621 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:29.932150 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:50:29.932707 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:50:29.932822 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:29.932926 (+ 104us) server_negotiation.cc:299] Negotiation successful
0806 02:50:29.933007 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:50:29.941855 19635 raft_consensus.cc:1261] T bcb19dc2950241f2af398bbd608947bc P b4a9b92b06f944ada0cae2be3bd2d89f [term 1 FOLLOWER]: Refusing update from remote peer 29e53c8ae6e44cc99ea148beecea4809: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:50:29.941855 19500 raft_consensus.cc:1261] T bcb19dc2950241f2af398bbd608947bc P 25e3e6227d894624831fb473dda136e6 [term 1 FOLLOWER]: Refusing update from remote peer 29e53c8ae6e44cc99ea148beecea4809: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:50:29.942878 19711 consensus_queue.cc:1035] T bcb19dc2950241f2af398bbd608947bc P 29e53c8ae6e44cc99ea148beecea4809 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4a9b92b06f944ada0cae2be3bd2d89f" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 38805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:50:29.943313 19707 consensus_queue.cc:1035] T bcb19dc2950241f2af398bbd608947bc P 29e53c8ae6e44cc99ea148beecea4809 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25e3e6227d894624831fb473dda136e6" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 41025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:50:29.972405 19219 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:50:29.972687 19219 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:50:29.976249 19219 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ad9f180a0f34a20b4c5574c3212ca97: Sending DeleteTablet for 3 replicas of tablet bcb19dc2950241f2af398bbd608947bc
I20260806 02:50:29.977451 19480 tablet_service.cc:1514] Processing DeleteTablet for tablet bcb19dc2950241f2af398bbd608947bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:29 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:47598
I20260806 02:50:29.977871 19615 tablet_service.cc:1514] Processing DeleteTablet for tablet bcb19dc2950241f2af398bbd608947bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:29 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:53866
I20260806 02:50:29.977939 19344 tablet_service.cc:1514] Processing DeleteTablet for tablet bcb19dc2950241f2af398bbd608947bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:29 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:33186
I20260806 02:50:29.983857 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 19281
I20260806 02:50:29.990185 19724 tablet_replica.cc:330] T bcb19dc2950241f2af398bbd608947bc P 25e3e6227d894624831fb473dda136e6: stopping tablet replica
I20260806 02:50:29.990515 19725 tablet_replica.cc:330] T bcb19dc2950241f2af398bbd608947bc P b4a9b92b06f944ada0cae2be3bd2d89f: stopping tablet replica
I20260806 02:50:29.990991 19724 raft_consensus.cc:2229] T bcb19dc2950241f2af398bbd608947bc P 25e3e6227d894624831fb473dda136e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:50:29.990990 19725 raft_consensus.cc:2229] T bcb19dc2950241f2af398bbd608947bc P b4a9b92b06f944ada0cae2be3bd2d89f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:50:29.991552 19725 raft_consensus.cc:2258] T bcb19dc2950241f2af398bbd608947bc P b4a9b92b06f944ada0cae2be3bd2d89f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:29.992185 19724 raft_consensus.cc:2258] T bcb19dc2950241f2af398bbd608947bc P 25e3e6227d894624831fb473dda136e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:29.993989 19725 ts_tablet_manager.cc:1905] T bcb19dc2950241f2af398bbd608947bc P b4a9b92b06f944ada0cae2be3bd2d89f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:29.994558 19724 ts_tablet_manager.cc:1905] T bcb19dc2950241f2af398bbd608947bc P 25e3e6227d894624831fb473dda136e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:29.995256 19726 tablet_replica.cc:330] T bcb19dc2950241f2af398bbd608947bc P 29e53c8ae6e44cc99ea148beecea4809: stopping tablet replica
I20260806 02:50:29.995728 19726 raft_consensus.cc:2229] T bcb19dc2950241f2af398bbd608947bc P 29e53c8ae6e44cc99ea148beecea4809 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:50:29.996405 19726 raft_consensus.cc:2258] T bcb19dc2950241f2af398bbd608947bc P 29e53c8ae6e44cc99ea148beecea4809 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:30.000669 19726 ts_tablet_manager.cc:1905] T bcb19dc2950241f2af398bbd608947bc P 29e53c8ae6e44cc99ea148beecea4809: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:30.003479 19724 ts_tablet_manager.cc:1918] T bcb19dc2950241f2af398bbd608947bc P 25e3e6227d894624831fb473dda136e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:50:30.003865 19724 log.cc:1196] T bcb19dc2950241f2af398bbd608947bc P 25e3e6227d894624831fb473dda136e6: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-1/wal/wals/bcb19dc2950241f2af398bbd608947bc
I20260806 02:50:30.004703 19724 ts_tablet_manager.cc:1939] T bcb19dc2950241f2af398bbd608947bc P 25e3e6227d894624831fb473dda136e6: Deleting consensus metadata
I20260806 02:50:30.007414 19205 catalog_manager.cc:4841] TS 25e3e6227d894624831fb473dda136e6 (127.1.210.66:41025): tablet bcb19dc2950241f2af398bbd608947bc (table test-table [id=b8632038629e40a78bc76d61044f4400]) successfully deleted
I20260806 02:50:30.008733 19725 ts_tablet_manager.cc:1918] T bcb19dc2950241f2af398bbd608947bc P b4a9b92b06f944ada0cae2be3bd2d89f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:50:30.008931 19725 log.cc:1196] T bcb19dc2950241f2af398bbd608947bc P b4a9b92b06f944ada0cae2be3bd2d89f: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-2/wal/wals/bcb19dc2950241f2af398bbd608947bc
I20260806 02:50:30.009495 19726 ts_tablet_manager.cc:1918] T bcb19dc2950241f2af398bbd608947bc P 29e53c8ae6e44cc99ea148beecea4809: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:50:30.009513 19725 ts_tablet_manager.cc:1939] T bcb19dc2950241f2af398bbd608947bc P b4a9b92b06f944ada0cae2be3bd2d89f: Deleting consensus metadata
I20260806 02:50:30.009693 19726 log.cc:1196] T bcb19dc2950241f2af398bbd608947bc P 29e53c8ae6e44cc99ea148beecea4809: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984304136196-1865-0/minicluster-data/ts-0/wal/wals/bcb19dc2950241f2af398bbd608947bc
I20260806 02:50:30.010218 19726 ts_tablet_manager.cc:1939] T bcb19dc2950241f2af398bbd608947bc P 29e53c8ae6e44cc99ea148beecea4809: Deleting consensus metadata
I20260806 02:50:30.011695 19205 catalog_manager.cc:4841] TS b4a9b92b06f944ada0cae2be3bd2d89f (127.1.210.67:38805): tablet bcb19dc2950241f2af398bbd608947bc (table test-table [id=b8632038629e40a78bc76d61044f4400]) successfully deleted
I20260806 02:50:30.012068 19205 catalog_manager.cc:4841] TS 29e53c8ae6e44cc99ea148beecea4809 (127.1.210.65:35991): tablet bcb19dc2950241f2af398bbd608947bc (table test-table [id=b8632038629e40a78bc76d61044f4400]) successfully deleted
Aug 06 02:50:30 dist-test-slave-8tdl krb5kdc[19173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.65@KRBTEST.COM
I20260806 02:50:30.026089 19697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:29.985195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33202 (local address 127.1.210.65:35991)
0806 02:50:29.985422 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:29.985429 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:29.999885 (+ 14456us) server_negotiation.cc:407] Connection header received
0806 02:50:30.000483 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:30.000489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:30.000577 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:30.000678 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:30.002781 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:30.004171 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:30.011579 (+ 7408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:30.012373 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:30.012490 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:30.018155 (+ 5665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:30.018176 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:30.018197 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:30.018255 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:30.022530 (+ 4275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:30.023564 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:30.023575 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:30.023578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:30.023789 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:30.024279 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:30.024284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:30.024288 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:30.024753 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:30.025099 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:30.025396 (+ 297us) server_negotiation.cc:299] Negotiation successful
0806 02:50:30.025656 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 02:50:30.026911 19405 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:50:30.133395 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 19281
I20260806 02:50:30.155994 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 19416
Aug 06 02:50:30 dist-test-slave-8tdl krb5kdc[19173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.66@KRBTEST.COM
I20260806 02:50:30.174034 19695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:30.157168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47602 (local address 127.1.210.66:41025)
0806 02:50:30.157502 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:30.157508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:30.157527 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:50:30.157587 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:30.157593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:30.157657 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:30.157750 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:30.159206 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:30.160043 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:30.161496 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:30.162003 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:30.162113 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:30.167095 (+ 4982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:30.167118 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:30.167136 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:30.167184 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:30.170377 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:30.171471 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:30.171482 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:30.171485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:30.171636 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:30.172226 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:30.172231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:30.172234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:30.172777 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:30.173034 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:30.173387 (+ 353us) server_negotiation.cc:299] Negotiation successful
0806 02:50:30.173611 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216}
I20260806 02:50:30.174822 19540 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:50:30.269742 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 19416
I20260806 02:50:30.288573 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 19552
Aug 06 02:50:30 dist-test-slave-8tdl krb5kdc[19173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.67@KRBTEST.COM
I20260806 02:50:30.306228 19699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:30.289496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53868 (local address 127.1.210.67:38805)
0806 02:50:30.289719 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:30.289725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:30.289748 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:50:30.289946 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:30.289952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:30.290042 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:30.290161 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:30.291738 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:30.292548 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:30.294042 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:30.294545 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:30.294788 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:30.300265 (+ 5477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:30.300283 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:30.300300 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:30.300340 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:30.303302 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:30.304081 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:30.304089 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:30.304091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:30.304235 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:30.304627 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:30.304637 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:30.304640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:30.305016 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:30.305317 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:30.305644 (+ 327us) server_negotiation.cc:299] Negotiation successful
0806 02:50:30.305867 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 02:50:30.306980 19676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:50:30.399269 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 19552
I20260806 02:50:30.417088 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 19186
I20260806 02:50:30.418445 19249 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:50:30.534600 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 19186
2026-08-06T02:50:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4664 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-06T02:50:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:30Z Disabled control of system clock
I20260806 02:50:30.633100 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:37853
--webserver_interface=127.1.210.126
--webserver_port=0
--builtin_ntp_servers=127.1.210.84:41347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:37853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 02:50:31.000948 19735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:31.001282 19735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:31.001446 19735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:31.009446 19735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:31.009588 19735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:31.009627 19735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:31.009739 19735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:31.009821 19735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:31.020140 19735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:41347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:37853
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:37853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:50:31.021639 19735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:31.023703 19735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:31.036337 19741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:31.036428 19743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:31.036404 19740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:31.037253 19735 server_base.cc:1034] running on GCE node
I20260806 02:50:31.038162 19735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:31.039749 19735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:31.041035 19735 hybrid_clock.cc:648] HybridClock initialized: now 1785984631040977 us; error 40 us; skew 500 ppm
I20260806 02:50:31.041478 19735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:50:31.050102 19735 webserver.cc:460] Webserver started at http://127.1.210.126:44839/ using document root <none> and password file <none>
I20260806 02:50:31.051106 19735 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:31.051241 19735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:31.051607 19735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:31.055013 19735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "79ea69519c9b4da8b1ddc12232b02b57"
format_stamp: "Formatted at 2026-08-06 02:50:31 on dist-test-slave-8tdl"
server_key: "1b0ab72c9818276e6bea7c87518fce8e"
server_key_iv: "45f89bc7048e25298cfeb5028b337b2c"
server_key_version: "encryptionkey@0"
I20260806 02:50:31.055908 19735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "79ea69519c9b4da8b1ddc12232b02b57"
format_stamp: "Formatted at 2026-08-06 02:50:31 on dist-test-slave-8tdl"
server_key: "1b0ab72c9818276e6bea7c87518fce8e"
server_key_iv: "45f89bc7048e25298cfeb5028b337b2c"
server_key_version: "encryptionkey@0"
I20260806 02:50:31.061823 19735 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 02:50:31.065958 19749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:31.067674 19735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 02:50:31.067903 19735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "79ea69519c9b4da8b1ddc12232b02b57"
format_stamp: "Formatted at 2026-08-06 02:50:31 on dist-test-slave-8tdl"
server_key: "1b0ab72c9818276e6bea7c87518fce8e"
server_key_iv: "45f89bc7048e25298cfeb5028b337b2c"
server_key_version: "encryptionkey@0"
I20260806 02:50:31.068152 19735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:31.089576 19735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:31.090559 19735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:31.090895 19735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:31.111560 19800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:37853 every 8 connection(s)
I20260806 02:50:31.111554 19735 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:37853
I20260806 02:50:31.113210 19735 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:31.115444 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 19735
I20260806 02:50:31.115653 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:50:31.116173 1865 external_mini_cluster.cc:1428] Setting key 31209d06b2320d4441c056ad7ba5e4a4
I20260806 02:50:31.118239 19801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:31.134016 19804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:31.118431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56992 (local address 127.1.210.126:37853)
0806 02:50:31.119696 (+ 1265us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:31.119712 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:31.119768 (+ 56us) server_negotiation.cc:407] Connection header received
0806 02:50:31.121946 (+ 2178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:31.121982 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:31.122812 (+ 830us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:31.123390 (+ 578us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:31.124030 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:31.126213 (+ 2183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:31.127905 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:31.129011 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:31.129211 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:31.130616 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:31.130627 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:31.130648 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:31.131561 (+ 913us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:31.131685 (+ 124us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:31.131719 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:31.131867 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:31.132165 (+ 298us) server_negotiation.cc:299] Negotiation successful
0806 02:50:31.132569 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":444,"threads_started":1}
I20260806 02:50:31.134644 19801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79ea69519c9b4da8b1ddc12232b02b57: Bootstrap starting.
I20260806 02:50:31.138106 19801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79ea69519c9b4da8b1ddc12232b02b57: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:31.139281 19801 log.cc:824] T 00000000000000000000000000000000 P 79ea69519c9b4da8b1ddc12232b02b57: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:31.142302 19801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79ea69519c9b4da8b1ddc12232b02b57: No bootstrap required, opened a new log
I20260806 02:50:31.149885 19801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79ea69519c9b4da8b1ddc12232b02b57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ea69519c9b4da8b1ddc12232b02b57" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 37853 } }
I20260806 02:50:31.150303 19801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79ea69519c9b4da8b1ddc12232b02b57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:31.150414 19801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79ea69519c9b4da8b1ddc12232b02b57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79ea69519c9b4da8b1ddc12232b02b57, State: Initialized, Role: FOLLOWER
I20260806 02:50:31.151161 19801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79ea69519c9b4da8b1ddc12232b02b57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ea69519c9b4da8b1ddc12232b02b57" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 37853 } }
I20260806 02:50:31.151479 19801 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79ea69519c9b4da8b1ddc12232b02b57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:50:31.151651 19801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79ea69519c9b4da8b1ddc12232b02b57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:50:31.151830 19801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79ea69519c9b4da8b1ddc12232b02b57 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:31.154075 19801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79ea69519c9b4da8b1ddc12232b02b57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ea69519c9b4da8b1ddc12232b02b57" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 37853 } }
I20260806 02:50:31.154616 19801 leader_election.cc:302] T 00000000000000000000000000000000 P 79ea69519c9b4da8b1ddc12232b02b57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79ea69519c9b4da8b1ddc12232b02b57; no voters:
I20260806 02:50:31.155376 19801 leader_election.cc:288] T 00000000000000000000000000000000 P 79ea69519c9b4da8b1ddc12232b02b57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:50:31.155563 19806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79ea69519c9b4da8b1ddc12232b02b57 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:31.156391 19806 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79ea69519c9b4da8b1ddc12232b02b57 [term 1 LEADER]: Becoming Leader. State: Replica: 79ea69519c9b4da8b1ddc12232b02b57, State: Running, Role: LEADER
I20260806 02:50:31.156949 19806 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79ea69519c9b4da8b1ddc12232b02b57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ea69519c9b4da8b1ddc12232b02b57" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 37853 } }
I20260806 02:50:31.157618 19801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79ea69519c9b4da8b1ddc12232b02b57 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:31.160081 19808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79ea69519c9b4da8b1ddc12232b02b57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79ea69519c9b4da8b1ddc12232b02b57. Latest consensus state: current_term: 1 leader_uuid: "79ea69519c9b4da8b1ddc12232b02b57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ea69519c9b4da8b1ddc12232b02b57" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 37853 } } }
I20260806 02:50:31.160507 19808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79ea69519c9b4da8b1ddc12232b02b57 [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:31.160954 19807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79ea69519c9b4da8b1ddc12232b02b57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79ea69519c9b4da8b1ddc12232b02b57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ea69519c9b4da8b1ddc12232b02b57" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 37853 } } }
I20260806 02:50:31.161268 19807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79ea69519c9b4da8b1ddc12232b02b57 [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:31.164075 19812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:50:31.169848 19812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:31.180130 19812 catalog_manager.cc:1347] Generated new cluster ID: d7caf3c961754155928ea9468d0e0dac
I20260806 02:50:31.180259 19812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:31.197742 19812 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:50:31.199776 19812 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:31.216099 19812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79ea69519c9b4da8b1ddc12232b02b57: Generated new TSK 0
I20260806 02:50:31.217336 19812 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:50:31.231626 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.65:0
--local_ip_for_outbound_sockets=127.1.210.65
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:37853
--builtin_ntp_servers=127.1.210.84:41347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 02:50:31.608691 19825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:31.609068 19825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:31.609236 19825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:31.617056 19825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:31.617231 19825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:31.617436 19825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.65
I20260806 02:50:31.626869 19825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:41347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:37853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.65
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:50:31.628312 19825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:31.630182 19825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:31.645372 19830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:31.645394 19833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:31.645346 19831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:31.646486 19825 server_base.cc:1034] running on GCE node
I20260806 02:50:31.647223 19825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:31.648409 19825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:31.649713 19825 hybrid_clock.cc:648] HybridClock initialized: now 1785984631649646 us; error 47 us; skew 500 ppm
I20260806 02:50:31.650246 19825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:50:31.652894 19825 webserver.cc:460] Webserver started at http://127.1.210.65:34853/ using document root <none> and password file <none>
I20260806 02:50:31.653630 19825 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:31.653776 19825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:31.654150 19825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:31.657434 19825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-0/data/instance:
uuid: "f34e58b1563848daa1e0e9daab2d3fc6"
format_stamp: "Formatted at 2026-08-06 02:50:31 on dist-test-slave-8tdl"
server_key: "b62a87849ac6861ea6290de479227058"
server_key_iv: "f03dcd2d8323be5d71d96d385bcd3154"
server_key_version: "encryptionkey@0"
I20260806 02:50:31.658339 19825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance:
uuid: "f34e58b1563848daa1e0e9daab2d3fc6"
format_stamp: "Formatted at 2026-08-06 02:50:31 on dist-test-slave-8tdl"
server_key: "b62a87849ac6861ea6290de479227058"
server_key_iv: "f03dcd2d8323be5d71d96d385bcd3154"
server_key_version: "encryptionkey@0"
I20260806 02:50:31.664016 19825 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 02:50:31.667948 19839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:31.669618 19825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:50:31.669809 19825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-0/wal
uuid: "f34e58b1563848daa1e0e9daab2d3fc6"
format_stamp: "Formatted at 2026-08-06 02:50:31 on dist-test-slave-8tdl"
server_key: "b62a87849ac6861ea6290de479227058"
server_key_iv: "f03dcd2d8323be5d71d96d385bcd3154"
server_key_version: "encryptionkey@0"
I20260806 02:50:31.670045 19825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:31.694489 19825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:31.695478 19825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:31.695806 19825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:31.696853 19825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:31.698537 19825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:31.698663 19825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:31.698843 19825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:31.698935 19825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:31.737489 19825 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.65:38381
I20260806 02:50:31.737581 19955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.65:38381 every 8 connection(s)
I20260806 02:50:31.738925 19825 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
I20260806 02:50:31.745965 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 19825
I20260806 02:50:31.746165 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance
I20260806 02:50:31.746651 1865 external_mini_cluster.cc:1428] Setting key 9c00adaeb0ecac348c0327ce53085a72
I20260806 02:50:31.749621 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.66:0
--local_ip_for_outbound_sockets=127.1.210.66
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:37853
--builtin_ntp_servers=127.1.210.84:41347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 02:50:31.752811 19958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:31.741810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:49591 (local address 127.1.210.126:37853)
0806 02:50:31.742364 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:31.742369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:31.743480 (+ 1111us) server_negotiation.cc:407] Connection header received
0806 02:50:31.745403 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:31.745409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:31.745512 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:31.745637 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:31.746942 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:31.747814 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:31.749429 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:31.749944 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:31.750107 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:31.751817 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:31.751822 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:31.751825 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:31.751845 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:31.751888 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:31.751901 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:31.752062 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:31.752322 (+ 260us) server_negotiation.cc:299] Negotiation successful
0806 02:50:31.752431 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":336,"threads_started":1}
I20260806 02:50:31.753926 19959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:31.742309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:37853 (local address 127.1.210.65:49591)
0806 02:50:31.743311 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:31.743350 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:31.745116 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:31.745878 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:31.745886 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:31.745987 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:31.746723 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:31.746768 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:31.747999 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:31.748011 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:31.749263 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:31.749276 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:31.750235 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:31.750245 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:31.750504 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:31.751362 (+ 858us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:50:31.751365 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:31.751422 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:50:31.751532 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:31.752056 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:31.752070 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:31.752081 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 02:50:31.752294 (+ 213us) client_negotiation.cc:240] Negotiation successful
0806 02:50:31.752796 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":328,"threads_started":1}
I20260806 02:50:31.755753 19956 heartbeater.cc:347] Connected to a master server at 127.1.210.126:37853
I20260806 02:50:31.756122 19956 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:31.757150 19956 heartbeater.cc:511] Master 127.1.210.126:37853 requested a full tablet report, sending...
I20260806 02:50:31.760047 19766 ts_manager.cc:194] Registered new tserver with Master: f34e58b1563848daa1e0e9daab2d3fc6 (127.1.210.65:38381)
I20260806 02:50:31.762009 19766 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.210.65:49591
W20260806 02:50:32.100446 19960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:32.100805 19960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:32.100976 19960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:32.108378 19960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:32.108536 19960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:32.108723 19960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.66
I20260806 02:50:32.118383 19960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:41347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:37853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.66
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:50:32.119784 19960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:32.121747 19960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:32.136272 19966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:32.136358 19965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:32.136862 19968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:32.137223 19960 server_base.cc:1034] running on GCE node
I20260806 02:50:32.137894 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:32.138947 19960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:32.140206 19960 hybrid_clock.cc:648] HybridClock initialized: now 1785984632140156 us; error 40 us; skew 500 ppm
I20260806 02:50:32.140583 19960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:50:32.143181 19960 webserver.cc:460] Webserver started at http://127.1.210.66:36701/ using document root <none> and password file <none>
I20260806 02:50:32.143919 19960 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:32.144057 19960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:32.144392 19960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:32.147471 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-1/data/instance:
uuid: "abcff10735d34ac6a92df393ecbea6b3"
format_stamp: "Formatted at 2026-08-06 02:50:32 on dist-test-slave-8tdl"
server_key: "8b9ef3514b752a90dcd70fab28712a50"
server_key_iv: "14c6b9e6fbb7b75f73f587b2c47d6eeb"
server_key_version: "encryptionkey@0"
I20260806 02:50:32.148319 19960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance:
uuid: "abcff10735d34ac6a92df393ecbea6b3"
format_stamp: "Formatted at 2026-08-06 02:50:32 on dist-test-slave-8tdl"
server_key: "8b9ef3514b752a90dcd70fab28712a50"
server_key_iv: "14c6b9e6fbb7b75f73f587b2c47d6eeb"
server_key_version: "encryptionkey@0"
I20260806 02:50:32.153874 19960 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 02:50:32.158104 19974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:32.159813 19960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:50:32.160024 19960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-1/wal
uuid: "abcff10735d34ac6a92df393ecbea6b3"
format_stamp: "Formatted at 2026-08-06 02:50:32 on dist-test-slave-8tdl"
server_key: "8b9ef3514b752a90dcd70fab28712a50"
server_key_iv: "14c6b9e6fbb7b75f73f587b2c47d6eeb"
server_key_version: "encryptionkey@0"
I20260806 02:50:32.160281 19960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:32.181114 19960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:32.182114 19960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:32.182456 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:32.183602 19960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:32.185197 19960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:32.185324 19960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:32.185467 19960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:32.185573 19960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:32.222514 19960 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.66:43821
I20260806 02:50:32.222609 20087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.66:43821 every 8 connection(s)
I20260806 02:50:32.223935 19960 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
I20260806 02:50:32.232860 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 19960
I20260806 02:50:32.233069 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance
I20260806 02:50:32.233657 1865 external_mini_cluster.cc:1428] Setting key a1b4d97b615f00baf6fd2581025b007a
I20260806 02:50:32.237004 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.67:0
--local_ip_for_outbound_sockets=127.1.210.67
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:37853
--builtin_ntp_servers=127.1.210.84:41347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 02:50:32.238229 19958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.226654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:46903 (local address 127.1.210.126:37853)
0806 02:50:32.226929 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:32.226936 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:32.228448 (+ 1512us) server_negotiation.cc:407] Connection header received
0806 02:50:32.230459 (+ 2011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:32.230464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:32.230545 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:32.230642 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:32.231879 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.232726 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.234949 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.235377 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.235501 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.237221 (+ 1720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:32.237230 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:32.237238 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:32.237266 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:32.237312 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:32.237331 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:32.237425 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:32.237676 (+ 251us) server_negotiation.cc:299] Negotiation successful
0806 02:50:32.237801 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260806 02:50:32.239460 20090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.227307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:37853 (local address 127.1.210.66:46903)
0806 02:50:32.228274 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:32.228316 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:32.230151 (+ 1835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:32.230951 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:32.230964 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:32.231074 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:32.231681 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:32.231699 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.232924 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.232931 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:32.234785 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:32.234799 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.235618 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.235625 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:32.235976 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.236794 (+ 818us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:50:32.236797 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:32.236837 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:50:32.236972 (+ 135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:32.237440 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:32.237449 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:32.237456 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 02:50:32.237609 (+ 153us) client_negotiation.cc:240] Negotiation successful
0806 02:50:32.238022 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":301,"threads_started":1}
I20260806 02:50:32.240976 20088 heartbeater.cc:347] Connected to a master server at 127.1.210.126:37853
I20260806 02:50:32.241322 20088 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:32.242146 20088 heartbeater.cc:511] Master 127.1.210.126:37853 requested a full tablet report, sending...
I20260806 02:50:32.243821 19766 ts_manager.cc:194] Registered new tserver with Master: abcff10735d34ac6a92df393ecbea6b3 (127.1.210.66:43821)
I20260806 02:50:32.244671 19766 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.210.66:46903
W20260806 02:50:32.584144 20091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:32.584484 20091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:32.584645 20091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:32.592535 20091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:32.592692 20091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:32.592870 20091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.67
I20260806 02:50:32.602823 20091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:41347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:37853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.67
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:50:32.604311 20091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:32.606302 20091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:32.620272 20096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:32.620206 20099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:32.620178 20097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:32.621351 20091 server_base.cc:1034] running on GCE node
I20260806 02:50:32.622059 20091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:32.623149 20091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:32.624395 20091 hybrid_clock.cc:648] HybridClock initialized: now 1785984632624337 us; error 52 us; skew 500 ppm
I20260806 02:50:32.624752 20091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:50:32.627281 20091 webserver.cc:460] Webserver started at http://127.1.210.67:35001/ using document root <none> and password file <none>
I20260806 02:50:32.627996 20091 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:32.628129 20091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:32.628455 20091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:32.631418 20091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-2/data/instance:
uuid: "83a2b22f79794c02bf6ee9a6c44a1edd"
format_stamp: "Formatted at 2026-08-06 02:50:32 on dist-test-slave-8tdl"
server_key: "ad40c4c64e794a24d6e1ea74aa07e95e"
server_key_iv: "2d46b70606c23fc1a721026257813053"
server_key_version: "encryptionkey@0"
I20260806 02:50:32.632242 20091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance:
uuid: "83a2b22f79794c02bf6ee9a6c44a1edd"
format_stamp: "Formatted at 2026-08-06 02:50:32 on dist-test-slave-8tdl"
server_key: "ad40c4c64e794a24d6e1ea74aa07e95e"
server_key_iv: "2d46b70606c23fc1a721026257813053"
server_key_version: "encryptionkey@0"
I20260806 02:50:32.637674 20091 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 02:50:32.641309 20105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:32.642956 20091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:50:32.643172 20091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-2/wal
uuid: "83a2b22f79794c02bf6ee9a6c44a1edd"
format_stamp: "Formatted at 2026-08-06 02:50:32 on dist-test-slave-8tdl"
server_key: "ad40c4c64e794a24d6e1ea74aa07e95e"
server_key_iv: "2d46b70606c23fc1a721026257813053"
server_key_version: "encryptionkey@0"
I20260806 02:50:32.643435 20091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:32.660001 20091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:32.660866 20091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:32.661202 20091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:32.662221 20091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:32.663967 20091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:32.664098 20091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:32.664225 20091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:32.664312 20091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:32.699296 20091 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.67:34499
I20260806 02:50:32.699353 20219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.67:34499 every 8 connection(s)
I20260806 02:50:32.700682 20091 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
I20260806 02:50:32.709251 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 20091
I20260806 02:50:32.709455 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance
I20260806 02:50:32.709928 1865 external_mini_cluster.cc:1428] Setting key 876aeeec6453600efccbc05e802dc374
I20260806 02:50:32.714283 19958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.703556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:39285 (local address 127.1.210.126:37853)
0806 02:50:32.703741 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:32.703746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:32.705124 (+ 1378us) server_negotiation.cc:407] Connection header received
0806 02:50:32.707067 (+ 1943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:32.707073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:32.707172 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:32.707286 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:32.708373 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.709159 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.711033 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.711553 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.711720 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.713434 (+ 1714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:32.713440 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:32.713444 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:32.713476 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:32.713508 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:32.713520 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:32.713614 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:32.713836 (+ 222us) server_negotiation.cc:299] Negotiation successful
0806 02:50:32.713935 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 02:50:32.715385 20222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.704076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:37853 (local address 127.1.210.67:39285)
0806 02:50:32.704952 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:32.705004 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:32.706786 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:32.707526 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:32.707534 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:32.707620 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:32.708208 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:32.708224 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.709308 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.709315 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:32.710874 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:32.710894 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.711804 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.711812 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:32.712082 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.713110 (+ 1028us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:50:32.713114 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:32.713167 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:50:32.713247 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:32.713615 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:32.713629 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:32.713638 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 02:50:32.713827 (+ 189us) client_negotiation.cc:240] Negotiation successful
0806 02:50:32.714250 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":259,"threads_started":1}
I20260806 02:50:32.717057 20220 heartbeater.cc:347] Connected to a master server at 127.1.210.126:37853
I20260806 02:50:32.717396 20220 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:32.718164 20220 heartbeater.cc:511] Master 127.1.210.126:37853 requested a full tablet report, sending...
I20260806 02:50:32.719715 19766 ts_manager.cc:194] Registered new tserver with Master: 83a2b22f79794c02bf6ee9a6c44a1edd (127.1.210.67:34499)
I20260806 02:50:32.720710 19766 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.210.67:39285
I20260806 02:50:32.726047 1865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:50:32.739010 19958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.730381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57002 (local address 127.1.210.126:37853)
0806 02:50:32.730618 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:32.730627 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:32.731213 (+ 586us) server_negotiation.cc:407] Connection header received
0806 02:50:32.731429 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:32.731435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:32.731515 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:32.731624 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:32.732157 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.732900 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.734304 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.734808 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.734964 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.736091 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:32.736097 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:32.736109 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:32.736128 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:32.736160 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:32.736172 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:32.736265 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:32.736417 (+ 152us) server_negotiation.cc:299] Negotiation successful
0806 02:50:32.736514 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260806 02:50:32.743557 19766 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57002:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:50:32.750885 19766 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:50:32.767853 19956 heartbeater.cc:503] Master 127.1.210.126:37853 was elected leader, sending a full tablet report...
I20260806 02:50:32.779749 20234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.766841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:34499 (local address 127.0.0.1:44274)
0806 02:50:32.767376 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:32.767394 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:32.767505 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:32.768858 (+ 1353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:32.768866 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:32.768911 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:32.769309 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:32.769322 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.772393 (+ 3071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.772401 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:32.775653 (+ 3252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:32.775679 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.778477 (+ 2798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.778489 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:32.778898 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.778985 (+ 87us) client_negotiation.cc:769] Sending connection context
0806 02:50:32.779142 (+ 157us) client_negotiation.cc:240] Negotiation successful
0806 02:50:32.779380 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":1583,"threads_started":1}
I20260806 02:50:32.779923 20236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.767457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44274 (local address 127.1.210.67:34499)
0806 02:50:32.768188 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:32.768197 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:32.768235 (+ 38us) server_negotiation.cc:407] Connection header received
0806 02:50:32.768341 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:32.768349 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:32.768624 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:32.769517 (+ 893us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:32.769836 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.772167 (+ 2331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.776075 (+ 3908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.778168 (+ 2093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.778369 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.778761 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:32.779307 (+ 546us) server_negotiation.cc:299] Negotiation successful
0806 02:50:32.779513 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":774,"threads_started":1}
I20260806 02:50:32.784873 20153 tablet_service.cc:1467] Processing CreateTablet for tablet e66f1dfd39f3442aa5f461adf3713af5 (DEFAULT_TABLE table=test-table [id=2c54e208b0cc485cb583de518155c40d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:32.787021 20153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e66f1dfd39f3442aa5f461adf3713af5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:32.787153 20232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.765388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:38381 (local address 127.0.0.1:59178)
0806 02:50:32.766301 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:32.766352 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:32.766541 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:32.780071 (+ 13530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:32.780077 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:32.780106 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:32.780578 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:32.780591 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.783000 (+ 2409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.783005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:32.784634 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:32.784649 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.786221 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.786229 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:32.786440 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.786477 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 02:50:32.786546 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 02:50:32.786727 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":77,"thread_start_us":497,"threads_started":1}
I20260806 02:50:32.788170 20237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.768063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59178 (local address 127.1.210.65:38381)
0806 02:50:32.779161 (+ 11098us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:32.779172 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:32.779208 (+ 36us) server_negotiation.cc:407] Connection header received
0806 02:50:32.779323 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:32.779330 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:32.779613 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:32.780859 (+ 1246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:32.781180 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.782863 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.784832 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.785986 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.787211 (+ 1225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.787527 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:32.787605 (+ 78us) server_negotiation.cc:299] Negotiation successful
0806 02:50:32.787788 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10877,"thread_start_us":11709,"threads_started":1}
I20260806 02:50:32.788544 20233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.766110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:43821 (local address 127.0.0.1:42764)
0806 02:50:32.766856 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:32.766879 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:32.767033 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:32.780096 (+ 13063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:32.780100 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:32.780116 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:32.780921 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:32.780937 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.782938 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.782944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:32.785452 (+ 2508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:32.785472 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.787699 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.787707 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:32.787917 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.787941 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 02:50:32.788049 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 02:50:32.788182 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":7,"spinlock_wait_cycles":1792,"thread_start_us":873,"threads_started":1}
I20260806 02:50:32.789640 20235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:32.766780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42764 (local address 127.1.210.66:43821)
0806 02:50:32.779205 (+ 12425us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:32.779212 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:32.779236 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:50:32.779321 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:32.779330 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:32.779613 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:32.779838 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:32.781122 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.782762 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.785688 (+ 2926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:32.787197 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:32.787334 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:32.787580 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:32.788098 (+ 518us) server_negotiation.cc:299] Negotiation successful
0806 02:50:32.788259 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12275,"thread_start_us":12158,"threads_started":1}
I20260806 02:50:32.791925 19890 tablet_service.cc:1467] Processing CreateTablet for tablet e66f1dfd39f3442aa5f461adf3713af5 (DEFAULT_TABLE table=test-table [id=2c54e208b0cc485cb583de518155c40d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:32.793581 19890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e66f1dfd39f3442aa5f461adf3713af5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:32.793659 20023 tablet_service.cc:1467] Processing CreateTablet for tablet e66f1dfd39f3442aa5f461adf3713af5 (DEFAULT_TABLE table=test-table [id=2c54e208b0cc485cb583de518155c40d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:32.795252 20023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e66f1dfd39f3442aa5f461adf3713af5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:32.803495 20238 tablet_bootstrap.cc:492] T e66f1dfd39f3442aa5f461adf3713af5 P 83a2b22f79794c02bf6ee9a6c44a1edd: Bootstrap starting.
I20260806 02:50:32.807577 20239 tablet_bootstrap.cc:492] T e66f1dfd39f3442aa5f461adf3713af5 P f34e58b1563848daa1e0e9daab2d3fc6: Bootstrap starting.
I20260806 02:50:32.808523 20238 tablet_bootstrap.cc:654] T e66f1dfd39f3442aa5f461adf3713af5 P 83a2b22f79794c02bf6ee9a6c44a1edd: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:32.809921 20238 log.cc:824] T e66f1dfd39f3442aa5f461adf3713af5 P 83a2b22f79794c02bf6ee9a6c44a1edd: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:32.811830 20239 tablet_bootstrap.cc:654] T e66f1dfd39f3442aa5f461adf3713af5 P f34e58b1563848daa1e0e9daab2d3fc6: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:32.813012 20238 tablet_bootstrap.cc:492] T e66f1dfd39f3442aa5f461adf3713af5 P 83a2b22f79794c02bf6ee9a6c44a1edd: No bootstrap required, opened a new log
I20260806 02:50:32.813074 20239 log.cc:824] T e66f1dfd39f3442aa5f461adf3713af5 P f34e58b1563848daa1e0e9daab2d3fc6: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:32.813271 20241 tablet_bootstrap.cc:492] T e66f1dfd39f3442aa5f461adf3713af5 P abcff10735d34ac6a92df393ecbea6b3: Bootstrap starting.
I20260806 02:50:32.813369 20238 ts_tablet_manager.cc:1397] T e66f1dfd39f3442aa5f461adf3713af5 P 83a2b22f79794c02bf6ee9a6c44a1edd: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260806 02:50:32.818499 20241 tablet_bootstrap.cc:654] T e66f1dfd39f3442aa5f461adf3713af5 P abcff10735d34ac6a92df393ecbea6b3: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:32.820016 20241 log.cc:824] T e66f1dfd39f3442aa5f461adf3713af5 P abcff10735d34ac6a92df393ecbea6b3: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:32.821533 20239 tablet_bootstrap.cc:492] T e66f1dfd39f3442aa5f461adf3713af5 P f34e58b1563848daa1e0e9daab2d3fc6: No bootstrap required, opened a new log
I20260806 02:50:32.821835 20239 ts_tablet_manager.cc:1397] T e66f1dfd39f3442aa5f461adf3713af5 P f34e58b1563848daa1e0e9daab2d3fc6: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260806 02:50:32.822155 20238 raft_consensus.cc:348] T e66f1dfd39f3442aa5f461adf3713af5 P 83a2b22f79794c02bf6ee9a6c44a1edd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34e58b1563848daa1e0e9daab2d3fc6" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 38381 } } peers { permanent_uuid: "83a2b22f79794c02bf6ee9a6c44a1edd" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 34499 } } peers { permanent_uuid: "abcff10735d34ac6a92df393ecbea6b3" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 43821 } }
I20260806 02:50:32.822624 20238 raft_consensus.cc:374] T e66f1dfd39f3442aa5f461adf3713af5 P 83a2b22f79794c02bf6ee9a6c44a1edd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:32.822780 20238 raft_consensus.cc:729] T e66f1dfd39f3442aa5f461adf3713af5 P 83a2b22f79794c02bf6ee9a6c44a1edd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83a2b22f79794c02bf6ee9a6c44a1edd, State: Initialized, Role: FOLLOWER
I20260806 02:50:32.823268 20241 tablet_bootstrap.cc:492] T e66f1dfd39f3442aa5f461adf3713af5 P abcff10735d34ac6a92df393ecbea6b3: No bootstrap required, opened a new log
I20260806 02:50:32.823560 20241 ts_tablet_manager.cc:1397] T e66f1dfd39f3442aa5f461adf3713af5 P abcff10735d34ac6a92df393ecbea6b3: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260806 02:50:32.823444 20238 consensus_queue.cc:260] T e66f1dfd39f3442aa5f461adf3713af5 P 83a2b22f79794c02bf6ee9a6c44a1edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34e58b1563848daa1e0e9daab2d3fc6" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 38381 } } peers { permanent_uuid: "83a2b22f79794c02bf6ee9a6c44a1edd" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 34499 } } peers { permanent_uuid: "abcff10735d34ac6a92df393ecbea6b3" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 43821 } }
I20260806 02:50:32.824679 20220 heartbeater.cc:503] Master 127.1.210.126:37853 was elected leader, sending a full tablet report...
I20260806 02:50:32.825258 20238 ts_tablet_manager.cc:1428] T e66f1dfd39f3442aa5f461adf3713af5 P 83a2b22f79794c02bf6ee9a6c44a1edd: Time spent starting tablet: real 0.012s user 0.006s sys 0.004s
I20260806 02:50:32.830474 20239 raft_consensus.cc:348] T e66f1dfd39f3442aa5f461adf3713af5 P f34e58b1563848daa1e0e9daab2d3fc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34e58b1563848daa1e0e9daab2d3fc6" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 38381 } } peers { permanent_uuid: "83a2b22f79794c02bf6ee9a6c44a1edd" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 34499 } } peers { permanent_uuid: "abcff10735d34ac6a92df393ecbea6b3" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 43821 } }
I20260806 02:50:32.830925 20239 raft_consensus.cc:374] T e66f1dfd39f3442aa5f461adf3713af5 P f34e58b1563848daa1e0e9daab2d3fc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:32.831044 20239 raft_consensus.cc:729] T e66f1dfd39f3442aa5f461adf3713af5 P f34e58b1563848daa1e0e9daab2d3fc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f34e58b1563848daa1e0e9daab2d3fc6, State: Initialized, Role: FOLLOWER
I20260806 02:50:32.831646 20239 consensus_queue.cc:260] T e66f1dfd39f3442aa5f461adf3713af5 P f34e58b1563848daa1e0e9daab2d3fc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34e58b1563848daa1e0e9daab2d3fc6" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 38381 } } peers { permanent_uuid: "83a2b22f79794c02bf6ee9a6c44a1edd" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 34499 } } peers { permanent_uuid: "abcff10735d34ac6a92df393ecbea6b3" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 43821 } }
I20260806 02:50:32.833235 20239 ts_tablet_manager.cc:1428] T e66f1dfd39f3442aa5f461adf3713af5 P f34e58b1563848daa1e0e9daab2d3fc6: Time spent starting tablet: real 0.011s user 0.013s sys 0.000s
I20260806 02:50:32.835004 20241 raft_consensus.cc:348] T e66f1dfd39f3442aa5f461adf3713af5 P abcff10735d34ac6a92df393ecbea6b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34e58b1563848daa1e0e9daab2d3fc6" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 38381 } } peers { permanent_uuid: "83a2b22f79794c02bf6ee9a6c44a1edd" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 34499 } } peers { permanent_uuid: "abcff10735d34ac6a92df393ecbea6b3" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 43821 } }
I20260806 02:50:32.835554 20241 raft_consensus.cc:374] T e66f1dfd39f3442aa5f461adf3713af5 P abcff10735d34ac6a92df393ecbea6b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:32.835695 20241 raft_consensus.cc:729] T e66f1dfd39f3442aa5f461adf3713af5 P abcff10735d34ac6a92df393ecbea6b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abcff10735d34ac6a92df393ecbea6b3, State: Initialized, Role: FOLLOWER
I20260806 02:50:32.836364 20241 consensus_queue.cc:260] T e66f1dfd39f3442aa5f461adf3713af5 P abcff10735d34ac6a92df393ecbea6b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34e58b1563848daa1e0e9daab2d3fc6" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 38381 } } peers { permanent_uuid: "83a2b22f79794c02bf6ee9a6c44a1edd" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 34499 } } peers { permanent_uuid: "abcff10735d34ac6a92df393ecbea6b3" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 43821 } }
I20260806 02:50:32.837579 20088 heartbeater.cc:503] Master 127.1.210.126:37853 was elected leader, sending a full tablet report...
I20260806 02:50:32.838376 20241 ts_tablet_manager.cc:1428] T e66f1dfd39f3442aa5f461adf3713af5 P abcff10735d34ac6a92df393ecbea6b3: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
W20260806 02:50:32.952551 20221 tablet.cc:2369] T e66f1dfd39f3442aa5f461adf3713af5 P 83a2b22f79794c02bf6ee9a6c44a1edd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:50:32.976202 20089 tablet.cc:2369] T e66f1dfd39f3442aa5f461adf3713af5 P abcff10735d34ac6a92df393ecbea6b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:50:32.992098 19957 tablet.cc:2369] T e66f1dfd39f3442aa5f461adf3713af5 P f34e58b1563848daa1e0e9daab2d3fc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:50:33.104589 20246 raft_consensus.cc:482] T e66f1dfd39f3442aa5f461adf3713af5 P abcff10735d34ac6a92df393ecbea6b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:50:33.105132 20246 raft_consensus.cc:504] T e66f1dfd39f3442aa5f461adf3713af5 P abcff10735d34ac6a92df393ecbea6b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34e58b1563848daa1e0e9daab2d3fc6" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 38381 } } peers { permanent_uuid: "83a2b22f79794c02bf6ee9a6c44a1edd" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 34499 } } peers { permanent_uuid: "abcff10735d34ac6a92df393ecbea6b3" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 43821 } }
I20260806 02:50:33.107399 20246 leader_election.cc:288] T e66f1dfd39f3442aa5f461adf3713af5 P abcff10735d34ac6a92df393ecbea6b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f34e58b1563848daa1e0e9daab2d3fc6 (127.1.210.65:38381), 83a2b22f79794c02bf6ee9a6c44a1edd (127.1.210.67:34499)
I20260806 02:50:33.115833 20236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:33.108892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:35185 (local address 127.1.210.67:34499)
0806 02:50:33.109065 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:33.109069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:33.109384 (+ 315us) server_negotiation.cc:407] Connection header received
0806 02:50:33.109575 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:33.109579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:33.109646 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:33.109717 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:33.110391 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:33.111492 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:33.113654 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:33.114761 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:33.114959 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:33.115035 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:33.115427 (+ 392us) server_negotiation.cc:299] Negotiation successful
0806 02:50:33.115570 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 02:50:33.115918 20248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:33.108593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:34499 (local address 127.1.210.66:35185)
0806 02:50:33.109276 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:33.109291 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:33.109431 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:33.109770 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:33.109775 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:33.109799 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:33.110216 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:33.110225 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:33.111647 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:33.111653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:33.113403 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:33.113422 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:33.115021 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:33.115028 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:33.115220 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:33.115259 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 02:50:33.115365 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 02:50:33.115515 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":28,"thread_start_us":379,"threads_started":1}
I20260806 02:50:33.116734 20237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:33.108020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:47661 (local address 127.1.210.65:38381)
0806 02:50:33.108318 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:33.108325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:33.108756 (+ 431us) server_negotiation.cc:407] Connection header received
0806 02:50:33.109226 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:33.109231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:33.109306 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:33.109421 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:33.110124 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:33.111337 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:33.114293 (+ 2956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:33.115507 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:33.115691 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:33.115757 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:33.116359 (+ 602us) server_negotiation.cc:299] Negotiation successful
0806 02:50:33.116502 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260806 02:50:33.116787 20247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:33.107705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:38381 (local address 127.1.210.66:47661)
0806 02:50:33.108567 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:33.108636 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:33.108822 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:33.109471 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:33.109479 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:33.109509 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:33.109921 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:33.109938 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:33.111558 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:33.111565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:33.114054 (+ 2489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:33.114072 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:33.115918 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:33.115925 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:33.116144 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:33.116176 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 02:50:33.116311 (+ 135us) client_negotiation.cc:240] Negotiation successful
0806 02:50:33.116469 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":384,"threads_started":1}
I20260806 02:50:33.117274 20174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e66f1dfd39f3442aa5f461adf3713af5" candidate_uuid: "abcff10735d34ac6a92df393ecbea6b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83a2b22f79794c02bf6ee9a6c44a1edd" is_pre_election: true
I20260806 02:50:33.117830 20174 raft_consensus.cc:2454] T e66f1dfd39f3442aa5f461adf3713af5 P 83a2b22f79794c02bf6ee9a6c44a1edd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abcff10735d34ac6a92df393ecbea6b3 in term 0.
I20260806 02:50:33.117738 19910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e66f1dfd39f3442aa5f461adf3713af5" candidate_uuid: "abcff10735d34ac6a92df393ecbea6b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f34e58b1563848daa1e0e9daab2d3fc6" is_pre_election: true
I20260806 02:50:33.118287 19910 raft_consensus.cc:2454] T e66f1dfd39f3442aa5f461adf3713af5 P f34e58b1563848daa1e0e9daab2d3fc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abcff10735d34ac6a92df393ecbea6b3 in term 0.
I20260806 02:50:33.118767 19976 leader_election.cc:302] T e66f1dfd39f3442aa5f461adf3713af5 P abcff10735d34ac6a92df393ecbea6b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83a2b22f79794c02bf6ee9a6c44a1edd, abcff10735d34ac6a92df393ecbea6b3; no voters:
I20260806 02:50:33.119266 20246 raft_consensus.cc:2781] T e66f1dfd39f3442aa5f461adf3713af5 P abcff10735d34ac6a92df393ecbea6b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:50:33.119460 20246 raft_consensus.cc:482] T e66f1dfd39f3442aa5f461adf3713af5 P abcff10735d34ac6a92df393ecbea6b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:50:33.119580 20246 raft_consensus.cc:3037] T e66f1dfd39f3442aa5f461adf3713af5 P abcff10735d34ac6a92df393ecbea6b3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:33.122002 20246 raft_consensus.cc:504] T e66f1dfd39f3442aa5f461adf3713af5 P abcff10735d34ac6a92df393ecbea6b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34e58b1563848daa1e0e9daab2d3fc6" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 38381 } } peers { permanent_uuid: "83a2b22f79794c02bf6ee9a6c44a1edd" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 34499 } } peers { permanent_uuid: "abcff10735d34ac6a92df393ecbea6b3" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 43821 } }
I20260806 02:50:33.122763 20246 leader_election.cc:288] T e66f1dfd39f3442aa5f461adf3713af5 P abcff10735d34ac6a92df393ecbea6b3 [CANDIDATE]: Term 1 election: Requested vote from peers f34e58b1563848daa1e0e9daab2d3fc6 (127.1.210.65:38381), 83a2b22f79794c02bf6ee9a6c44a1edd (127.1.210.67:34499)
I20260806 02:50:33.123368 19910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e66f1dfd39f3442aa5f461adf3713af5" candidate_uuid: "abcff10735d34ac6a92df393ecbea6b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f34e58b1563848daa1e0e9daab2d3fc6"
I20260806 02:50:33.123473 20174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e66f1dfd39f3442aa5f461adf3713af5" candidate_uuid: "abcff10735d34ac6a92df393ecbea6b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83a2b22f79794c02bf6ee9a6c44a1edd"
I20260806 02:50:33.123644 19910 raft_consensus.cc:3037] T e66f1dfd39f3442aa5f461adf3713af5 P f34e58b1563848daa1e0e9daab2d3fc6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:33.123703 20174 raft_consensus.cc:3037] T e66f1dfd39f3442aa5f461adf3713af5 P 83a2b22f79794c02bf6ee9a6c44a1edd [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:33.126104 19910 raft_consensus.cc:2454] T e66f1dfd39f3442aa5f461adf3713af5 P f34e58b1563848daa1e0e9daab2d3fc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abcff10735d34ac6a92df393ecbea6b3 in term 1.
I20260806 02:50:33.126649 19976 leader_election.cc:302] T e66f1dfd39f3442aa5f461adf3713af5 P abcff10735d34ac6a92df393ecbea6b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abcff10735d34ac6a92df393ecbea6b3, f34e58b1563848daa1e0e9daab2d3fc6; no voters:
I20260806 02:50:33.127108 20246 raft_consensus.cc:2781] T e66f1dfd39f3442aa5f461adf3713af5 P abcff10735d34ac6a92df393ecbea6b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:33.127122 20174 raft_consensus.cc:2454] T e66f1dfd39f3442aa5f461adf3713af5 P 83a2b22f79794c02bf6ee9a6c44a1edd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abcff10735d34ac6a92df393ecbea6b3 in term 1.
I20260806 02:50:33.127696 20246 raft_consensus.cc:686] T e66f1dfd39f3442aa5f461adf3713af5 P abcff10735d34ac6a92df393ecbea6b3 [term 1 LEADER]: Becoming Leader. State: Replica: abcff10735d34ac6a92df393ecbea6b3, State: Running, Role: LEADER
I20260806 02:50:33.128388 20246 consensus_queue.cc:237] T e66f1dfd39f3442aa5f461adf3713af5 P abcff10735d34ac6a92df393ecbea6b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34e58b1563848daa1e0e9daab2d3fc6" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 38381 } } peers { permanent_uuid: "83a2b22f79794c02bf6ee9a6c44a1edd" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 34499 } } peers { permanent_uuid: "abcff10735d34ac6a92df393ecbea6b3" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 43821 } }
I20260806 02:50:33.133689 19766 catalog_manager.cc:5495] T e66f1dfd39f3442aa5f461adf3713af5 P abcff10735d34ac6a92df393ecbea6b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to abcff10735d34ac6a92df393ecbea6b3 (127.1.210.66). New cstate: current_term: 1 leader_uuid: "abcff10735d34ac6a92df393ecbea6b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34e58b1563848daa1e0e9daab2d3fc6" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 38381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83a2b22f79794c02bf6ee9a6c44a1edd" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 34499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abcff10735d34ac6a92df393ecbea6b3" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 43821 } health_report { overall_health: HEALTHY } } }
I20260806 02:50:33.151547 20235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:33.145309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42770 (local address 127.1.210.66:43821)
0806 02:50:33.145562 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:33.145567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:33.145586 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:50:33.145800 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:33.145804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:33.145917 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:33.146046 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:50:33.146459 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:33.147478 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:33.149265 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:33.149689 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:33.149788 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:33.150260 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:50:33.150885 (+ 625us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:50:33.151009 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:33.151159 (+ 150us) server_negotiation.cc:299] Negotiation successful
0806 02:50:33.151230 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 02:50:33.161139 19910 raft_consensus.cc:1261] T e66f1dfd39f3442aa5f461adf3713af5 P f34e58b1563848daa1e0e9daab2d3fc6 [term 1 FOLLOWER]: Refusing update from remote peer abcff10735d34ac6a92df393ecbea6b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:50:33.161139 20174 raft_consensus.cc:1261] T e66f1dfd39f3442aa5f461adf3713af5 P 83a2b22f79794c02bf6ee9a6c44a1edd [term 1 FOLLOWER]: Refusing update from remote peer abcff10735d34ac6a92df393ecbea6b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:50:33.162386 20249 consensus_queue.cc:1035] T e66f1dfd39f3442aa5f461adf3713af5 P abcff10735d34ac6a92df393ecbea6b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f34e58b1563848daa1e0e9daab2d3fc6" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 38381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:50:33.162889 20246 consensus_queue.cc:1035] T e66f1dfd39f3442aa5f461adf3713af5 P abcff10735d34ac6a92df393ecbea6b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83a2b22f79794c02bf6ee9a6c44a1edd" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 34499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:50:33.177038 20252 mvcc.cc:205] Tried to move back new op lower bound from 7315393057415684096 to 7315393057298608128. Current Snapshot: MvccSnapshot[applied={T|T < 7315393057415684096}]
I20260806 02:50:33.191632 19766 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:50:33.191954 19766 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:50:33.196311 19766 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 79ea69519c9b4da8b1ddc12232b02b57: Sending DeleteTablet for 3 replicas of tablet e66f1dfd39f3442aa5f461adf3713af5
I20260806 02:50:33.197830 20023 tablet_service.cc:1514] Processing DeleteTablet for tablet e66f1dfd39f3442aa5f461adf3713af5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:33 UTC) from {username='slave'} at 127.0.0.1:42764
I20260806 02:50:33.198100 19890 tablet_service.cc:1514] Processing DeleteTablet for tablet e66f1dfd39f3442aa5f461adf3713af5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:33 UTC) from {username='slave'} at 127.0.0.1:59178
I20260806 02:50:33.198900 20153 tablet_service.cc:1514] Processing DeleteTablet for tablet e66f1dfd39f3442aa5f461adf3713af5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:33 UTC) from {username='slave'} at 127.0.0.1:44274
I20260806 02:50:33.199134 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 19825
I20260806 02:50:33.198933 20262 tablet_replica.cc:330] T e66f1dfd39f3442aa5f461adf3713af5 P abcff10735d34ac6a92df393ecbea6b3: stopping tablet replica
I20260806 02:50:33.200043 20262 raft_consensus.cc:2229] T e66f1dfd39f3442aa5f461adf3713af5 P abcff10735d34ac6a92df393ecbea6b3 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:50:33.200796 20262 raft_consensus.cc:2258] T e66f1dfd39f3442aa5f461adf3713af5 P abcff10735d34ac6a92df393ecbea6b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:33.202811 20263 tablet_replica.cc:330] T e66f1dfd39f3442aa5f461adf3713af5 P f34e58b1563848daa1e0e9daab2d3fc6: stopping tablet replica
I20260806 02:50:33.203387 20263 raft_consensus.cc:2229] T e66f1dfd39f3442aa5f461adf3713af5 P f34e58b1563848daa1e0e9daab2d3fc6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:50:33.204129 20263 raft_consensus.cc:2258] T e66f1dfd39f3442aa5f461adf3713af5 P f34e58b1563848daa1e0e9daab2d3fc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:33.206138 20262 ts_tablet_manager.cc:1905] T e66f1dfd39f3442aa5f461adf3713af5 P abcff10735d34ac6a92df393ecbea6b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:33.209064 20263 ts_tablet_manager.cc:1905] T e66f1dfd39f3442aa5f461adf3713af5 P f34e58b1563848daa1e0e9daab2d3fc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:33.212065 20237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:33.200467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59184 (local address 127.1.210.65:38381)
0806 02:50:33.200670 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:33.200678 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:33.202041 (+ 1363us) server_negotiation.cc:407] Connection header received
0806 02:50:33.202387 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:33.202396 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:33.202538 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:33.202644 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:33.203579 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:33.205004 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:33.207700 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:33.208383 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:33.208514 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:33.210174 (+ 1660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:33.210186 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:33.210204 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:33.211071 (+ 867us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:33.211140 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:33.211172 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:33.211258 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:33.211522 (+ 264us) server_negotiation.cc:299] Negotiation successful
0806 02:50:33.211652 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 02:50:33.212913 20264 tablet_replica.cc:330] T e66f1dfd39f3442aa5f461adf3713af5 P 83a2b22f79794c02bf6ee9a6c44a1edd: stopping tablet replica
I20260806 02:50:33.213342 19951 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:50:33.213593 20264 raft_consensus.cc:2229] T e66f1dfd39f3442aa5f461adf3713af5 P 83a2b22f79794c02bf6ee9a6c44a1edd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:50:33.214219 20264 raft_consensus.cc:2258] T e66f1dfd39f3442aa5f461adf3713af5 P 83a2b22f79794c02bf6ee9a6c44a1edd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:33.217631 20264 ts_tablet_manager.cc:1905] T e66f1dfd39f3442aa5f461adf3713af5 P 83a2b22f79794c02bf6ee9a6c44a1edd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:33.218403 20262 ts_tablet_manager.cc:1918] T e66f1dfd39f3442aa5f461adf3713af5 P abcff10735d34ac6a92df393ecbea6b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:50:33.218652 20262 log.cc:1196] T e66f1dfd39f3442aa5f461adf3713af5 P abcff10735d34ac6a92df393ecbea6b3: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-1/wal/wals/e66f1dfd39f3442aa5f461adf3713af5
I20260806 02:50:33.219411 20262 ts_tablet_manager.cc:1939] T e66f1dfd39f3442aa5f461adf3713af5 P abcff10735d34ac6a92df393ecbea6b3: Deleting consensus metadata
I20260806 02:50:33.221922 19752 catalog_manager.cc:4841] TS abcff10735d34ac6a92df393ecbea6b3 (127.1.210.66:43821): tablet e66f1dfd39f3442aa5f461adf3713af5 (table test-table [id=2c54e208b0cc485cb583de518155c40d]) successfully deleted
I20260806 02:50:33.225720 20264 ts_tablet_manager.cc:1918] T e66f1dfd39f3442aa5f461adf3713af5 P 83a2b22f79794c02bf6ee9a6c44a1edd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:50:33.226006 20264 log.cc:1196] T e66f1dfd39f3442aa5f461adf3713af5 P 83a2b22f79794c02bf6ee9a6c44a1edd: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984304136196-1865-0/minicluster-data/ts-2/wal/wals/e66f1dfd39f3442aa5f461adf3713af5
I20260806 02:50:33.226635 20264 ts_tablet_manager.cc:1939] T e66f1dfd39f3442aa5f461adf3713af5 P 83a2b22f79794c02bf6ee9a6c44a1edd: Deleting consensus metadata
I20260806 02:50:33.228765 19751 catalog_manager.cc:4841] TS 83a2b22f79794c02bf6ee9a6c44a1edd (127.1.210.67:34499): tablet e66f1dfd39f3442aa5f461adf3713af5 (table test-table [id=2c54e208b0cc485cb583de518155c40d]) successfully deleted
I20260806 02:50:33.331023 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 19825
I20260806 02:50:33.352530 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 19960
W20260806 02:50:33.352568 19751 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.210.65:38381 (error 108)
I20260806 02:50:33.355278 20268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:33.354208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:38381 (local address 127.0.0.1:59186)
0806 02:50:33.354894 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:33.355050 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:38381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":360,"threads_started":1}
W20260806 02:50:33.355638 19751 catalog_manager.cc:4584] TS f34e58b1563848daa1e0e9daab2d3fc6 (127.1.210.65:38381): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e66f1dfd39f3442aa5f461adf3713af5: Network error: Client connection negotiation failed: client connection to 127.1.210.65:38381: connect: Connection refused (error 111)
I20260806 02:50:33.361604 20235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:33.353849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42772 (local address 127.1.210.66:43821)
0806 02:50:33.354026 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:33.354030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:33.354047 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:50:33.354109 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:33.354112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:33.354201 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:33.354323 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:33.354891 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:33.356073 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:33.358210 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:33.358667 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:33.358860 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:33.359994 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:33.360004 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:33.360017 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:33.360678 (+ 661us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:33.360735 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:33.360763 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:33.360848 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:33.361079 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 02:50:33.361178 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:50:33.362398 20083 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:50:33.409654 20268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:33.409037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:38381 (local address 127.0.0.1:59198)
0806 02:50:33.409326 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:33.409459 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:38381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260806 02:50:33.410959 20268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:33.410471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:38381 (local address 127.0.0.1:59210)
0806 02:50:33.410655 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:33.410778 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:38381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 02:50:33.471750 20268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:33.471203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:38381 (local address 127.0.0.1:59226)
0806 02:50:33.471459 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:33.471571 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:38381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260806 02:50:33.473129 20268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:33.472662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:38381 (local address 127.0.0.1:59228)
0806 02:50:33.472840 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:33.472935 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:38381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 02:50:33.481037 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 19960
I20260806 02:50:33.503077 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 20091
I20260806 02:50:33.511178 20236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:33.503913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44288 (local address 127.1.210.67:34499)
0806 02:50:33.504105 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:33.504110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:33.504130 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:50:33.504311 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:33.504315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:33.504417 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:33.504510 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:33.504984 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:33.505830 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:33.507432 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:33.507852 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:33.507967 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:33.509366 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:33.509377 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:33.509393 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:33.510197 (+ 804us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:33.510271 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:33.510297 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:33.510412 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:33.510616 (+ 204us) server_negotiation.cc:299] Negotiation successful
0806 02:50:33.510807 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 02:50:33.512072 20215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:50:33.553939 20268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:33.553327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:38381 (local address 127.0.0.1:59230)
0806 02:50:33.553628 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:33.553747 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:38381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260806 02:50:33.555249 20268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:33.554757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:38381 (local address 127.0.0.1:59234)
0806 02:50:33.555005 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:33.555087 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:38381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260806 02:50:33.605095 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 20091
I20260806 02:50:33.623754 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 19735
I20260806 02:50:33.625043 19796 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:50:33.742971 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 19735
2026-08-06T02:50:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3198 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:50:33 dist-test-slave-8tdl krb5kdc[20274](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:50:33 dist-test-slave-8tdl krb5kdc[20274](info): set up 2 sockets
Aug 06 02:50:33 dist-test-slave-8tdl krb5kdc[20274](info): commencing operation
krb5kdc: starting...
W20260806 02:50:34.886400 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.003s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:50:35 dist-test-slave-8tdl krb5kdc[20274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:50:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
I20260806 02:50:35.203047 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:35767
--webserver_interface=127.1.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:42641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:35767
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:35.551689 20287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:35.552007 20287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:35.552155 20287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:35.559988 20287 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:35.560137 20287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:35.560245 20287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:35.560341 20287 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:35.560407 20287 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:35.571064 20287 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:42641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:35767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:35767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:50:35.572595 20287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:35.574065 20287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:35.586786 20292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:35.586717 20295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:35.586820 20293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:35.588439 20287 server_base.cc:1034] running on GCE node
I20260806 02:50:35.589398 20287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:35.591135 20287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:35.592361 20287 hybrid_clock.cc:648] HybridClock initialized: now 1785984635592308 us; error 48 us; skew 500 ppm
I20260806 02:50:35.592751 20287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:50:35 dist-test-slave-8tdl krb5kdc[20274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984635, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:35.596125 20287 init.cc:377] Logged in from keytab as kudu/127.1.210.126@KRBTEST.COM (short username kudu)
I20260806 02:50:35.597817 20287 webserver.cc:460] Webserver started at http://127.1.210.126:33825/ using document root <none> and password file <none>
I20260806 02:50:35.598547 20287 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:35.598748 20287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:35.599084 20287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:35.602231 20287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "ee88f33cffcf409896239db4039a8301"
format_stamp: "Formatted at 2026-08-06 02:50:35 on dist-test-slave-8tdl"
server_key: "24e5092fa1c5b1201c28bbfad4258c92"
server_key_iv: "ef99336983aedcc2f317264d6c286249"
server_key_version: "encryptionkey@0"
I20260806 02:50:35.603106 20287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "ee88f33cffcf409896239db4039a8301"
format_stamp: "Formatted at 2026-08-06 02:50:35 on dist-test-slave-8tdl"
server_key: "24e5092fa1c5b1201c28bbfad4258c92"
server_key_iv: "ef99336983aedcc2f317264d6c286249"
server_key_version: "encryptionkey@0"
I20260806 02:50:35.608733 20287 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260806 02:50:35.612625 20302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:35.614233 20287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:50:35.614449 20287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "ee88f33cffcf409896239db4039a8301"
format_stamp: "Formatted at 2026-08-06 02:50:35 on dist-test-slave-8tdl"
server_key: "24e5092fa1c5b1201c28bbfad4258c92"
server_key_iv: "ef99336983aedcc2f317264d6c286249"
server_key_version: "encryptionkey@0"
I20260806 02:50:35.614671 20287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:35.635737 20287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:35.640318 20287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:35.640658 20287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:35.661304 20287 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:35767
I20260806 02:50:35.661327 20354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:35767 every 8 connection(s)
I20260806 02:50:35.662837 20287 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:35.664606 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 20287
I20260806 02:50:35.664811 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:50:35.665398 1865 external_mini_cluster.cc:1428] Setting key 0ecf23058bef9b0a360291d0fe0fa6b8
I20260806 02:50:35.667936 20355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 02:50:35 dist-test-slave-8tdl krb5kdc[20274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:50:35.681923 20355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee88f33cffcf409896239db4039a8301: Bootstrap starting.
I20260806 02:50:35.685726 20355 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee88f33cffcf409896239db4039a8301: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:35.687039 20355 log.cc:824] T 00000000000000000000000000000000 P ee88f33cffcf409896239db4039a8301: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:35.690894 20355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee88f33cffcf409896239db4039a8301: No bootstrap required, opened a new log
I20260806 02:50:35.690960 20358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:35.667440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34238 (local address 127.1.210.126:35767)
0806 02:50:35.668292 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:35.668305 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:35.668349 (+ 44us) server_negotiation.cc:407] Connection header received
0806 02:50:35.670086 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:35.670108 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:35.670528 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:35.670952 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:35.672961 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.674359 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.676242 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:35.676939 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:35.677097 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:35.682267 (+ 5170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:35.682275 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:35.682290 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:35.682329 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:35.685746 (+ 3417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:35.686562 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:35.686569 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:35.686581 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:35.686852 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:35.687352 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:35.687358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:35.687361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:35.688150 (+ 789us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:35.688409 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:35.689032 (+ 623us) server_negotiation.cc:299] Negotiation successful
0806 02:50:35.689525 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":398,"threads_started":1}
I20260806 02:50:35.699018 20355 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee88f33cffcf409896239db4039a8301 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee88f33cffcf409896239db4039a8301" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 35767 } }
I20260806 02:50:35.699563 20355 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee88f33cffcf409896239db4039a8301 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:35.699725 20355 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee88f33cffcf409896239db4039a8301 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee88f33cffcf409896239db4039a8301, State: Initialized, Role: FOLLOWER
I20260806 02:50:35.700459 20355 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee88f33cffcf409896239db4039a8301 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee88f33cffcf409896239db4039a8301" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 35767 } }
I20260806 02:50:35.700740 20355 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee88f33cffcf409896239db4039a8301 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:50:35.700892 20355 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee88f33cffcf409896239db4039a8301 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:50:35.701066 20355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee88f33cffcf409896239db4039a8301 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:35.703475 20355 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee88f33cffcf409896239db4039a8301 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee88f33cffcf409896239db4039a8301" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 35767 } }
I20260806 02:50:35.704020 20355 leader_election.cc:302] T 00000000000000000000000000000000 P ee88f33cffcf409896239db4039a8301 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee88f33cffcf409896239db4039a8301; no voters:
I20260806 02:50:35.704720 20355 leader_election.cc:288] T 00000000000000000000000000000000 P ee88f33cffcf409896239db4039a8301 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:50:35.704873 20360 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee88f33cffcf409896239db4039a8301 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:35.705765 20360 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee88f33cffcf409896239db4039a8301 [term 1 LEADER]: Becoming Leader. State: Replica: ee88f33cffcf409896239db4039a8301, State: Running, Role: LEADER
I20260806 02:50:35.706357 20360 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee88f33cffcf409896239db4039a8301 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee88f33cffcf409896239db4039a8301" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 35767 } }
I20260806 02:50:35.707297 20355 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee88f33cffcf409896239db4039a8301 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:35.709580 20361 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee88f33cffcf409896239db4039a8301 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee88f33cffcf409896239db4039a8301" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee88f33cffcf409896239db4039a8301" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 35767 } } }
I20260806 02:50:35.710100 20361 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee88f33cffcf409896239db4039a8301 [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:35.709606 20362 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee88f33cffcf409896239db4039a8301 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee88f33cffcf409896239db4039a8301. Latest consensus state: current_term: 1 leader_uuid: "ee88f33cffcf409896239db4039a8301" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee88f33cffcf409896239db4039a8301" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 35767 } } }
I20260806 02:50:35.711252 20362 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee88f33cffcf409896239db4039a8301 [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:35.715325 20368 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:50:35.720742 20368 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:35.734719 20368 catalog_manager.cc:1347] Generated new cluster ID: 3f3b1fcbd07448528556c7ceca0a4529
I20260806 02:50:35.734848 20368 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:35.746855 20368 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:50:35.748319 20368 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:35.760150 20368 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee88f33cffcf409896239db4039a8301: Generated new TSK 0
I20260806 02:50:35.761114 20368 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab.
Entry for principal kudu/127.1.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab.
I20260806 02:50:35.836308 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.65:0
--local_ip_for_outbound_sockets=127.1.210.65
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:35767
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:42641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:36.201902 20382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:36.202208 20382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:36.202308 20382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:36.210203 20382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:36.210343 20382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:36.210466 20382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.65
I20260806 02:50:36.221275 20382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:42641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:35767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.65
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:50:36.222575 20382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:36.223995 20382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:36.238585 20387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:36.238552 20388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:36.238659 20390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:36.239642 20382 server_base.cc:1034] running on GCE node
I20260806 02:50:36.240458 20382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:36.241628 20382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:36.242846 20382 hybrid_clock.cc:648] HybridClock initialized: now 1785984636242797 us; error 41 us; skew 500 ppm
I20260806 02:50:36.243294 20382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:50:36 dist-test-slave-8tdl krb5kdc[20274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984636, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:36.252004 20382 init.cc:377] Logged in from keytab as kudu/127.1.210.65@KRBTEST.COM (short username kudu)
I20260806 02:50:36.253778 20382 webserver.cc:460] Webserver started at http://127.1.210.65:35687/ using document root <none> and password file <none>
I20260806 02:50:36.254549 20382 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:36.254663 20382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:36.255059 20382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:36.258383 20382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-0/data/instance:
uuid: "7ec4af4579204039a1e4c072c0435792"
format_stamp: "Formatted at 2026-08-06 02:50:36 on dist-test-slave-8tdl"
server_key: "36d01f59d5d7922b62babb47470d5975"
server_key_iv: "49a171b7c7fd68b1e4642e9d0b4abfc8"
server_key_version: "encryptionkey@0"
I20260806 02:50:36.259666 20382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance:
uuid: "7ec4af4579204039a1e4c072c0435792"
format_stamp: "Formatted at 2026-08-06 02:50:36 on dist-test-slave-8tdl"
server_key: "36d01f59d5d7922b62babb47470d5975"
server_key_iv: "49a171b7c7fd68b1e4642e9d0b4abfc8"
server_key_version: "encryptionkey@0"
I20260806 02:50:36.265870 20382 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260806 02:50:36.270227 20397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:36.271927 20382 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:50:36.272152 20382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-0/wal
uuid: "7ec4af4579204039a1e4c072c0435792"
format_stamp: "Formatted at 2026-08-06 02:50:36 on dist-test-slave-8tdl"
server_key: "36d01f59d5d7922b62babb47470d5975"
server_key_iv: "49a171b7c7fd68b1e4642e9d0b4abfc8"
server_key_version: "encryptionkey@0"
I20260806 02:50:36.272377 20382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:36.303687 20382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:36.309214 20382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:36.309543 20382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:36.310703 20382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:36.312191 20382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:36.312268 20382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:36.312397 20382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:36.312461 20382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:36.349980 20382 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.65:37619
I20260806 02:50:36.350101 20510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.65:37619 every 8 connection(s)
I20260806 02:50:36.351456 20382 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
I20260806 02:50:36.358605 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 20382
I20260806 02:50:36.358868 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance
I20260806 02:50:36.359373 1865 external_mini_cluster.cc:1428] Setting key 1cfa3573fffdb8014890916d6d27735f
Aug 06 02:50:36 dist-test-slave-8tdl krb5kdc[20274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984636, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:50:36.373220 20513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:36.354350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:44405 (local address 127.1.210.126:35767)
0806 02:50:36.354888 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:36.354893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:36.355883 (+ 990us) server_negotiation.cc:407] Connection header received
0806 02:50:36.357805 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:36.357811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:36.357907 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:36.358031 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:36.359836 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:36.360723 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:36.362501 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:36.362988 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:36.363128 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:36.366851 (+ 3723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:36.366857 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:36.366860 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:36.366895 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:36.369860 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:36.370743 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:36.370748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:36.370751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:36.370880 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:36.371360 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:36.371364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:36.371366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:36.371650 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:36.371815 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:36.372581 (+ 766us) server_negotiation.cc:299] Negotiation successful
0806 02:50:36.372835 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":276,"threads_started":1}
I20260806 02:50:36.374259 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:36.354999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:35767 (local address 127.1.210.65:44405)
0806 02:50:36.355715 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:36.355760 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:36.357527 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:36.358311 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:36.358320 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:36.359001 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:36.359660 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:36.359680 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:36.360911 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:36.360915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:36.362352 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:36.362365 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:36.363240 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:36.363248 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:36.363489 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:36.364263 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:36.364286 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:36.366571 (+ 2285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:36.370079 (+ 3508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:36.370088 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:36.370104 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:36.370579 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:36.371005 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:36.371017 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:36.371021 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:36.371225 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:36.371786 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:36.371795 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:36.372173 (+ 378us) client_negotiation.cc:769] Sending connection context
0806 02:50:36.372449 (+ 276us) client_negotiation.cc:240] Negotiation successful
0806 02:50:36.373058 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":283,"threads_started":1}
I20260806 02:50:36.376032 20511 heartbeater.cc:347] Connected to a master server at 127.1.210.126:35767
I20260806 02:50:36.376426 20511 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:36.377489 20511 heartbeater.cc:511] Master 127.1.210.126:35767 requested a full tablet report, sending...
I20260806 02:50:36.380287 20320 ts_manager.cc:194] Registered new tserver with Master: 7ec4af4579204039a1e4c072c0435792 (127.1.210.65:37619)
I20260806 02:50:36.382817 20320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.65@KRBTEST.COM'} at 127.1.210.65:44405
WARNING: no policy specified for kudu/127.1.210.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab.
Entry for principal kudu/127.1.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab.
I20260806 02:50:36.435277 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.66:0
--local_ip_for_outbound_sockets=127.1.210.66
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:35767
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:42641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:36.784313 20517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:36.784600 20517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:36.784761 20517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:36.792568 20517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:36.792697 20517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:36.792881 20517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.66
I20260806 02:50:36.803134 20517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:42641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:35767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.66
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:50:36.804651 20517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:36.806043 20517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:36.820773 20522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:36.820915 20525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:36.820947 20523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:36.821863 20517 server_base.cc:1034] running on GCE node
I20260806 02:50:36.822487 20517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:36.823604 20517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:36.824851 20517 hybrid_clock.cc:648] HybridClock initialized: now 1785984636824782 us; error 63 us; skew 500 ppm
I20260806 02:50:36.825203 20517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:50:36 dist-test-slave-8tdl krb5kdc[20274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984636, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:36.828748 20517 init.cc:377] Logged in from keytab as kudu/127.1.210.66@KRBTEST.COM (short username kudu)
I20260806 02:50:36.830539 20517 webserver.cc:460] Webserver started at http://127.1.210.66:32993/ using document root <none> and password file <none>
I20260806 02:50:36.831332 20517 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:36.831487 20517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:36.831813 20517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:36.834885 20517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-1/data/instance:
uuid: "cb0691f7254e43ee833bba5171b544ee"
format_stamp: "Formatted at 2026-08-06 02:50:36 on dist-test-slave-8tdl"
server_key: "7f8cda5f5fc5c264bab629bd90a69fca"
server_key_iv: "a2f98be76d6f800ddd0feae2f4d6293e"
server_key_version: "encryptionkey@0"
I20260806 02:50:36.835721 20517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance:
uuid: "cb0691f7254e43ee833bba5171b544ee"
format_stamp: "Formatted at 2026-08-06 02:50:36 on dist-test-slave-8tdl"
server_key: "7f8cda5f5fc5c264bab629bd90a69fca"
server_key_iv: "a2f98be76d6f800ddd0feae2f4d6293e"
server_key_version: "encryptionkey@0"
I20260806 02:50:36.841040 20517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 02:50:36.844717 20532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:36.846344 20517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 02:50:36.846536 20517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-1/wal
uuid: "cb0691f7254e43ee833bba5171b544ee"
format_stamp: "Formatted at 2026-08-06 02:50:36 on dist-test-slave-8tdl"
server_key: "7f8cda5f5fc5c264bab629bd90a69fca"
server_key_iv: "a2f98be76d6f800ddd0feae2f4d6293e"
server_key_version: "encryptionkey@0"
I20260806 02:50:36.846809 20517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:36.865264 20517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:36.869872 20517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:36.870209 20517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:36.871277 20517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:36.872735 20517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:36.872845 20517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:36.873000 20517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:36.873080 20517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:36.908918 20517 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.66:37131
I20260806 02:50:36.909044 20648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.66:37131 every 8 connection(s)
I20260806 02:50:36.910458 20517 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
I20260806 02:50:36.917135 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 20517
I20260806 02:50:36.917356 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance
I20260806 02:50:36.917943 1865 external_mini_cluster.cc:1428] Setting key 55a6f07575efe84e909c0397ba8cb5e0
Aug 06 02:50:36 dist-test-slave-8tdl krb5kdc[20274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984636, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:50:36.934052 20651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:36.913146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:51193 (local address 127.1.210.126:35767)
0806 02:50:36.913592 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:36.913597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:36.914709 (+ 1112us) server_negotiation.cc:407] Connection header received
0806 02:50:36.916622 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:36.916627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:36.916700 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:36.916801 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:36.919045 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:36.920102 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:36.921714 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:36.922160 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:36.922302 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:36.926787 (+ 4485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:36.926798 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:36.926805 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:36.926856 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:36.930090 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:36.931227 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:36.931232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:36.931235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:36.931386 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:36.931901 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:36.931904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:36.931906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:36.932194 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:36.932348 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:36.933286 (+ 938us) server_negotiation.cc:299] Negotiation successful
0806 02:50:36.933578 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":229,"threads_started":1}
I20260806 02:50:36.934835 20652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:36.913781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:35767 (local address 127.1.210.66:51193)
0806 02:50:36.914509 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:36.914549 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:36.916337 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:36.917151 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:36.917167 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:36.917985 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:36.918850 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:36.918872 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:36.920281 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:36.920285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:36.921551 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:36.921565 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:36.922404 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:36.922411 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:36.922650 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:36.923503 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:36.923532 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:36.926325 (+ 2793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:36.930283 (+ 3958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:36.930296 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:36.930319 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:36.931085 (+ 766us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:36.931502 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:36.931509 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:36.931516 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:36.931784 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:36.932348 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:36.932360 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:36.932867 (+ 507us) client_negotiation.cc:769] Sending connection context
0806 02:50:36.933117 (+ 250us) client_negotiation.cc:240] Negotiation successful
0806 02:50:36.933549 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":292,"threads_started":1}
I20260806 02:50:36.936520 20649 heartbeater.cc:347] Connected to a master server at 127.1.210.126:35767
I20260806 02:50:36.936892 20649 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:36.937872 20649 heartbeater.cc:511] Master 127.1.210.126:35767 requested a full tablet report, sending...
I20260806 02:50:36.939925 20320 ts_manager.cc:194] Registered new tserver with Master: cb0691f7254e43ee833bba5171b544ee (127.1.210.66:37131)
I20260806 02:50:36.940889 20320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.66@KRBTEST.COM'} at 127.1.210.66:51193
WARNING: no policy specified for kudu/127.1.210.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab.
Entry for principal kudu/127.1.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab.
I20260806 02:50:36.990742 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.67:0
--local_ip_for_outbound_sockets=127.1.210.67
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:35767
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:42641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:37.341660 20655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:37.342010 20655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:37.342188 20655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:37.349525 20655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:37.349689 20655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:37.349877 20655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.67
I20260806 02:50:37.360152 20655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:42641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:35767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.67
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:50:37.361501 20655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:37.362905 20655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:37.376431 20663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:37.376294 20661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:37.376438 20660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:37.377488 20655 server_base.cc:1034] running on GCE node
I20260806 02:50:37.378099 20655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:37.379086 20655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:37.380318 20655 hybrid_clock.cc:648] HybridClock initialized: now 1785984637380245 us; error 57 us; skew 500 ppm
I20260806 02:50:37.380719 20655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:50:37 dist-test-slave-8tdl krb5kdc[20274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984637, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:37.384505 20655 init.cc:377] Logged in from keytab as kudu/127.1.210.67@KRBTEST.COM (short username kudu)
I20260806 02:50:37.386009 20655 webserver.cc:460] Webserver started at http://127.1.210.67:34107/ using document root <none> and password file <none>
I20260806 02:50:37.386755 20655 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:37.386881 20655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:37.387166 20655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:37.387312 20511 heartbeater.cc:503] Master 127.1.210.126:35767 was elected leader, sending a full tablet report...
I20260806 02:50:37.390615 20655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-2/data/instance:
uuid: "ce770dd6e4d7499aa8a57fbca40d2579"
format_stamp: "Formatted at 2026-08-06 02:50:37 on dist-test-slave-8tdl"
server_key: "5dc06c59c90fe50d9d862f0b0bba24c6"
server_key_iv: "cb806023141aaec1f0e50d74196977ea"
server_key_version: "encryptionkey@0"
I20260806 02:50:37.391551 20655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance:
uuid: "ce770dd6e4d7499aa8a57fbca40d2579"
format_stamp: "Formatted at 2026-08-06 02:50:37 on dist-test-slave-8tdl"
server_key: "5dc06c59c90fe50d9d862f0b0bba24c6"
server_key_iv: "cb806023141aaec1f0e50d74196977ea"
server_key_version: "encryptionkey@0"
I20260806 02:50:37.397325 20655 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 02:50:37.401218 20670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:37.402768 20655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 02:50:37.402964 20655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-2/wal
uuid: "ce770dd6e4d7499aa8a57fbca40d2579"
format_stamp: "Formatted at 2026-08-06 02:50:37 on dist-test-slave-8tdl"
server_key: "5dc06c59c90fe50d9d862f0b0bba24c6"
server_key_iv: "cb806023141aaec1f0e50d74196977ea"
server_key_version: "encryptionkey@0"
I20260806 02:50:37.403188 20655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:37.442291 20655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:37.447022 20655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:37.447350 20655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:37.448393 20655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:37.449873 20655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:37.449988 20655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:37.450145 20655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:37.450217 20655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:37.485549 20655 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.67:45775
I20260806 02:50:37.485647 20785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.67:45775 every 8 connection(s)
I20260806 02:50:37.487056 20655 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
I20260806 02:50:37.493415 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 20655
I20260806 02:50:37.493637 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance
I20260806 02:50:37.494203 1865 external_mini_cluster.cc:1428] Setting key 77ea4673e325cf27b7ac052121900eec
Aug 06 02:50:37 dist-test-slave-8tdl krb5kdc[20274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984637, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:50:37.510242 20788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:37.490199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:44011 (local address 127.1.210.126:35767)
0806 02:50:37.490779 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:37.490787 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:37.491809 (+ 1022us) server_negotiation.cc:407] Connection header received
0806 02:50:37.494211 (+ 2402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:37.494215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:37.494331 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:37.494440 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:37.496592 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.497702 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.499403 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.499814 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.499956 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:37.503669 (+ 3713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:37.503674 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:37.503677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:37.503712 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:37.506334 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:37.507333 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:37.507338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:37.507341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:37.507505 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:37.508033 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:37.508038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:37.508041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:37.508390 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:37.508575 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:37.509560 (+ 985us) server_negotiation.cc:299] Negotiation successful
0806 02:50:37.509831 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":240,"threads_started":1}
I20260806 02:50:37.511348 20789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:37.490820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:35767 (local address 127.1.210.67:44011)
0806 02:50:37.491632 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:37.491673 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:37.493832 (+ 2159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:37.494765 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:37.494785 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:37.495523 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:37.496410 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:37.496443 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.497864 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.497869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:37.499235 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:37.499247 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.500055 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.500062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:37.500318 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:37.501080 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:37.501104 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:37.503421 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:37.506531 (+ 3110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:37.506539 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:37.506553 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:37.507203 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:37.507650 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:37.507655 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:37.507658 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:37.507893 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:37.508539 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:37.508546 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:37.509058 (+ 512us) client_negotiation.cc:769] Sending connection context
0806 02:50:37.509436 (+ 378us) client_negotiation.cc:240] Negotiation successful
0806 02:50:37.510036 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":320,"threads_started":1}
I20260806 02:50:37.512655 20786 heartbeater.cc:347] Connected to a master server at 127.1.210.126:35767
I20260806 02:50:37.512980 20786 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:37.513811 20786 heartbeater.cc:511] Master 127.1.210.126:35767 requested a full tablet report, sending...
I20260806 02:50:37.515324 20320 ts_manager.cc:194] Registered new tserver with Master: ce770dd6e4d7499aa8a57fbca40d2579 (127.1.210.67:45775)
I20260806 02:50:37.516095 20320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.67@KRBTEST.COM'} at 127.1.210.67:44011
I20260806 02:50:37.522665 1865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:50:37.541795 20788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:37.527269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34244 (local address 127.1.210.126:35767)
0806 02:50:37.527545 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:37.527553 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:37.527895 (+ 342us) server_negotiation.cc:407] Connection header received
0806 02:50:37.528177 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:37.528182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:37.528296 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:37.528422 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:37.529922 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.530762 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.532124 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.532561 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.532699 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:37.535776 (+ 3077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:37.535784 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:37.535788 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:37.535836 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:37.538483 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:37.539444 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:37.539449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:37.539453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:37.539602 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:37.540087 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:37.540093 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:37.540096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:37.540434 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:37.540736 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:37.541130 (+ 394us) server_negotiation.cc:299] Negotiation successful
0806 02:50:37.541344 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260806 02:50:37.546211 20320 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:50:37.555256 20320 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:50:37.585572 20801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:37.569968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:37619 (local address 127.0.0.1:41812)
0806 02:50:37.572741 (+ 2773us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:37.572793 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:37.573406 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:37.574741 (+ 1335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:37.574752 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:37.574806 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:37.575447 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:37.575472 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.578056 (+ 2584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.578065 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:37.581487 (+ 3422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:37.581521 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.584476 (+ 2955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.584486 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:37.584729 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:37.584795 (+ 66us) client_negotiation.cc:769] Sending connection context
0806 02:50:37.584922 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 02:50:37.585124 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2423,"mutex_wait_us":5,"thread_start_us":3617,"threads_started":1}
I20260806 02:50:37.585601 20799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:37.570009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41812 (local address 127.1.210.65:37619)
0806 02:50:37.571596 (+ 1587us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:37.571609 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:37.572934 (+ 1325us) server_negotiation.cc:407] Connection header received
0806 02:50:37.573760 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:37.573769 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:37.574105 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:37.574274 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:37.575717 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.577812 (+ 2095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.581750 (+ 3938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.583755 (+ 2005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.583902 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:37.584189 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:37.585001 (+ 812us) server_negotiation.cc:299] Negotiation successful
0806 02:50:37.585235 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1332,"thread_start_us":1123,"threads_started":1}
I20260806 02:50:37.587747 20800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:37.571903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:37131 (local address 127.0.0.1:50846)
0806 02:50:37.573896 (+ 1993us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:37.573915 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:37.574140 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:37.575079 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:37.575082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:37.575104 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:37.575473 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:37.575480 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.579899 (+ 4419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.579908 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:37.584430 (+ 4522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:37.584445 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.586885 (+ 2440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.586892 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:37.587091 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:37.587122 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 02:50:37.587249 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 02:50:37.587409 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1877,"thread_start_us":326,"threads_started":1}
I20260806 02:50:37.587893 20802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:37.572270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50846 (local address 127.1.210.66:37131)
0806 02:50:37.573441 (+ 1171us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:37.573450 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:37.574023 (+ 573us) server_negotiation.cc:407] Connection header received
0806 02:50:37.574362 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:37.574370 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:37.574626 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:37.574836 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:37.575668 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.577877 (+ 2209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.584660 (+ 6783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.586312 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.586443 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:37.586644 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:37.587341 (+ 697us) server_negotiation.cc:299] Negotiation successful
0806 02:50:37.587523 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":988,"thread_start_us":883,"threads_started":1}
I20260806 02:50:37.590361 20445 tablet_service.cc:1467] Processing CreateTablet for tablet 66834c6ca26b45ad891689c87807227d (DEFAULT_TABLE table=test-table [id=5c34ca88fec04b408a27bd9e5416dfe2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:37.591187 20581 tablet_service.cc:1467] Processing CreateTablet for tablet 66834c6ca26b45ad891689c87807227d (DEFAULT_TABLE table=test-table [id=5c34ca88fec04b408a27bd9e5416dfe2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:37.591737 20445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66834c6ca26b45ad891689c87807227d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:37.592520 20581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66834c6ca26b45ad891689c87807227d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:37.595837 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:37.575798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:45775 (local address 127.0.0.1:37472)
0806 02:50:37.584224 (+ 8426us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:37.584245 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:37.584493 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:37.585559 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:37.585565 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:37.585597 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:37.585964 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:37.585975 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.587800 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.587805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:37.592638 (+ 4833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:37.592656 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.594813 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.594820 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:37.595101 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:37.595152 (+ 51us) client_negotiation.cc:769] Sending connection context
0806 02:50:37.595252 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 02:50:37.595403 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8293,"mutex_wait_us":48,"thread_start_us":8008,"threads_started":1}
I20260806 02:50:37.596562 20803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:37.574256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37472 (local address 127.1.210.67:45775)
0806 02:50:37.583496 (+ 9240us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:37.583505 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:37.584367 (+ 862us) server_negotiation.cc:407] Connection header received
0806 02:50:37.584762 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:37.584772 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:37.585056 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:37.585269 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:37.586222 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.587574 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.592838 (+ 5264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.594295 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.594445 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:37.595940 (+ 1495us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:37.596037 (+ 97us) server_negotiation.cc:299] Negotiation successful
0806 02:50:37.596208 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9051,"thread_start_us":8874,"threads_started":1}
I20260806 02:50:37.600553 20719 tablet_service.cc:1467] Processing CreateTablet for tablet 66834c6ca26b45ad891689c87807227d (DEFAULT_TABLE table=test-table [id=5c34ca88fec04b408a27bd9e5416dfe2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:37.602138 20719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66834c6ca26b45ad891689c87807227d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:37.605479 20805 tablet_bootstrap.cc:492] T 66834c6ca26b45ad891689c87807227d P 7ec4af4579204039a1e4c072c0435792: Bootstrap starting.
I20260806 02:50:37.607668 20806 tablet_bootstrap.cc:492] T 66834c6ca26b45ad891689c87807227d P cb0691f7254e43ee833bba5171b544ee: Bootstrap starting.
I20260806 02:50:37.609611 20805 tablet_bootstrap.cc:654] T 66834c6ca26b45ad891689c87807227d P 7ec4af4579204039a1e4c072c0435792: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:37.611042 20805 log.cc:824] T 66834c6ca26b45ad891689c87807227d P 7ec4af4579204039a1e4c072c0435792: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:37.611505 20806 tablet_bootstrap.cc:654] T 66834c6ca26b45ad891689c87807227d P cb0691f7254e43ee833bba5171b544ee: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:37.613075 20806 log.cc:824] T 66834c6ca26b45ad891689c87807227d P cb0691f7254e43ee833bba5171b544ee: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:37.616441 20806 tablet_bootstrap.cc:492] T 66834c6ca26b45ad891689c87807227d P cb0691f7254e43ee833bba5171b544ee: No bootstrap required, opened a new log
I20260806 02:50:37.616763 20805 tablet_bootstrap.cc:492] T 66834c6ca26b45ad891689c87807227d P 7ec4af4579204039a1e4c072c0435792: No bootstrap required, opened a new log
I20260806 02:50:37.616827 20809 tablet_bootstrap.cc:492] T 66834c6ca26b45ad891689c87807227d P ce770dd6e4d7499aa8a57fbca40d2579: Bootstrap starting.
I20260806 02:50:37.616811 20806 ts_tablet_manager.cc:1397] T 66834c6ca26b45ad891689c87807227d P cb0691f7254e43ee833bba5171b544ee: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.006s
I20260806 02:50:37.617115 20805 ts_tablet_manager.cc:1397] T 66834c6ca26b45ad891689c87807227d P 7ec4af4579204039a1e4c072c0435792: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260806 02:50:37.620597 20809 tablet_bootstrap.cc:654] T 66834c6ca26b45ad891689c87807227d P ce770dd6e4d7499aa8a57fbca40d2579: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:37.621907 20809 log.cc:824] T 66834c6ca26b45ad891689c87807227d P ce770dd6e4d7499aa8a57fbca40d2579: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:37.625752 20809 tablet_bootstrap.cc:492] T 66834c6ca26b45ad891689c87807227d P ce770dd6e4d7499aa8a57fbca40d2579: No bootstrap required, opened a new log
I20260806 02:50:37.626044 20809 ts_tablet_manager.cc:1397] T 66834c6ca26b45ad891689c87807227d P ce770dd6e4d7499aa8a57fbca40d2579: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20260806 02:50:37.626327 20805 raft_consensus.cc:348] T 66834c6ca26b45ad891689c87807227d P 7ec4af4579204039a1e4c072c0435792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec4af4579204039a1e4c072c0435792" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 37619 } } peers { permanent_uuid: "ce770dd6e4d7499aa8a57fbca40d2579" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45775 } } peers { permanent_uuid: "cb0691f7254e43ee833bba5171b544ee" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37131 } }
I20260806 02:50:37.626748 20805 raft_consensus.cc:374] T 66834c6ca26b45ad891689c87807227d P 7ec4af4579204039a1e4c072c0435792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:37.626869 20805 raft_consensus.cc:729] T 66834c6ca26b45ad891689c87807227d P 7ec4af4579204039a1e4c072c0435792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec4af4579204039a1e4c072c0435792, State: Initialized, Role: FOLLOWER
I20260806 02:50:37.627565 20805 consensus_queue.cc:260] T 66834c6ca26b45ad891689c87807227d P 7ec4af4579204039a1e4c072c0435792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec4af4579204039a1e4c072c0435792" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 37619 } } peers { permanent_uuid: "ce770dd6e4d7499aa8a57fbca40d2579" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45775 } } peers { permanent_uuid: "cb0691f7254e43ee833bba5171b544ee" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37131 } }
I20260806 02:50:37.628226 20806 raft_consensus.cc:348] T 66834c6ca26b45ad891689c87807227d P cb0691f7254e43ee833bba5171b544ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec4af4579204039a1e4c072c0435792" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 37619 } } peers { permanent_uuid: "ce770dd6e4d7499aa8a57fbca40d2579" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45775 } } peers { permanent_uuid: "cb0691f7254e43ee833bba5171b544ee" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37131 } }
I20260806 02:50:37.628794 20806 raft_consensus.cc:374] T 66834c6ca26b45ad891689c87807227d P cb0691f7254e43ee833bba5171b544ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:37.628932 20806 raft_consensus.cc:729] T 66834c6ca26b45ad891689c87807227d P cb0691f7254e43ee833bba5171b544ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb0691f7254e43ee833bba5171b544ee, State: Initialized, Role: FOLLOWER
I20260806 02:50:37.629503 20805 ts_tablet_manager.cc:1428] T 66834c6ca26b45ad891689c87807227d P 7ec4af4579204039a1e4c072c0435792: Time spent starting tablet: real 0.012s user 0.006s sys 0.004s
I20260806 02:50:37.629658 20806 consensus_queue.cc:260] T 66834c6ca26b45ad891689c87807227d P cb0691f7254e43ee833bba5171b544ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec4af4579204039a1e4c072c0435792" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 37619 } } peers { permanent_uuid: "ce770dd6e4d7499aa8a57fbca40d2579" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45775 } } peers { permanent_uuid: "cb0691f7254e43ee833bba5171b544ee" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37131 } }
I20260806 02:50:37.631808 20806 ts_tablet_manager.cc:1428] T 66834c6ca26b45ad891689c87807227d P cb0691f7254e43ee833bba5171b544ee: Time spent starting tablet: real 0.015s user 0.010s sys 0.003s
I20260806 02:50:37.632786 20649 heartbeater.cc:503] Master 127.1.210.126:35767 was elected leader, sending a full tablet report...
I20260806 02:50:37.636297 20809 raft_consensus.cc:348] T 66834c6ca26b45ad891689c87807227d P ce770dd6e4d7499aa8a57fbca40d2579 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec4af4579204039a1e4c072c0435792" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 37619 } } peers { permanent_uuid: "ce770dd6e4d7499aa8a57fbca40d2579" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45775 } } peers { permanent_uuid: "cb0691f7254e43ee833bba5171b544ee" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37131 } }
I20260806 02:50:37.636736 20809 raft_consensus.cc:374] T 66834c6ca26b45ad891689c87807227d P ce770dd6e4d7499aa8a57fbca40d2579 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:37.636859 20809 raft_consensus.cc:729] T 66834c6ca26b45ad891689c87807227d P ce770dd6e4d7499aa8a57fbca40d2579 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce770dd6e4d7499aa8a57fbca40d2579, State: Initialized, Role: FOLLOWER
I20260806 02:50:37.637460 20809 consensus_queue.cc:260] T 66834c6ca26b45ad891689c87807227d P ce770dd6e4d7499aa8a57fbca40d2579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec4af4579204039a1e4c072c0435792" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 37619 } } peers { permanent_uuid: "ce770dd6e4d7499aa8a57fbca40d2579" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45775 } } peers { permanent_uuid: "cb0691f7254e43ee833bba5171b544ee" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37131 } }
I20260806 02:50:37.638551 20786 heartbeater.cc:503] Master 127.1.210.126:35767 was elected leader, sending a full tablet report...
I20260806 02:50:37.639030 20809 ts_tablet_manager.cc:1428] T 66834c6ca26b45ad891689c87807227d P ce770dd6e4d7499aa8a57fbca40d2579: Time spent starting tablet: real 0.013s user 0.007s sys 0.007s
W20260806 02:50:37.662945 20650 tablet.cc:2369] T 66834c6ca26b45ad891689c87807227d P cb0691f7254e43ee833bba5171b544ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:50:37.713034 20811 raft_consensus.cc:482] T 66834c6ca26b45ad891689c87807227d P 7ec4af4579204039a1e4c072c0435792 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:50:37.713339 20811 raft_consensus.cc:504] T 66834c6ca26b45ad891689c87807227d P 7ec4af4579204039a1e4c072c0435792 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec4af4579204039a1e4c072c0435792" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 37619 } } peers { permanent_uuid: "ce770dd6e4d7499aa8a57fbca40d2579" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45775 } } peers { permanent_uuid: "cb0691f7254e43ee833bba5171b544ee" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37131 } }
I20260806 02:50:37.715345 20811 leader_election.cc:288] T 66834c6ca26b45ad891689c87807227d P 7ec4af4579204039a1e4c072c0435792 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce770dd6e4d7499aa8a57fbca40d2579 (127.1.210.67:45775), cb0691f7254e43ee833bba5171b544ee (127.1.210.66:37131)
I20260806 02:50:37.723695 20802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:37.716121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:41177 (local address 127.1.210.66:37131)
0806 02:50:37.716333 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:37.716338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:37.716935 (+ 597us) server_negotiation.cc:407] Connection header received
0806 02:50:37.717112 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:37.717116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:37.717213 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:37.717310 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:37.718001 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.718911 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.721468 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.722338 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.722444 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:37.722499 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:37.723126 (+ 627us) server_negotiation.cc:299] Negotiation successful
0806 02:50:37.723311 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 02:50:37.723747 20815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:37.715711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:37131 (local address 127.1.210.65:41177)
0806 02:50:37.716806 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:37.716830 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:37.716973 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:37.717351 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:37.717354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:37.717387 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:37.717814 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:37.717825 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.719042 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.719045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:37.721247 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:37.721268 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.722723 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.722732 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:37.722917 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:37.722949 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 02:50:37.723058 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 02:50:37.723204 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":944,"mutex_wait_us":56,"thread_start_us":778,"threads_started":1}
I20260806 02:50:37.724517 20803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:37.715699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:51015 (local address 127.1.210.67:45775)
0806 02:50:37.715961 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:37.715971 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:37.716766 (+ 795us) server_negotiation.cc:407] Connection header received
0806 02:50:37.716978 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:37.716985 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:37.717071 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:37.717160 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:37.717821 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.718622 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.721468 (+ 2846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.723029 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.723276 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:37.723374 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:37.724045 (+ 671us) server_negotiation.cc:299] Negotiation successful
0806 02:50:37.724215 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 02:50:37.725333 20814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:37.715553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:45775 (local address 127.1.210.65:51015)
0806 02:50:37.716609 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:37.716639 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:37.716825 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:37.717204 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:37.717209 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:37.717233 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:37.717649 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:37.717670 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.718840 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.718864 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:37.721250 (+ 2386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:37.721270 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.723555 (+ 2285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.723565 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:37.723819 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:37.723862 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 02:50:37.724722 (+ 860us) client_negotiation.cc:240] Negotiation successful
0806 02:50:37.724894 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"mutex_wait_us":135,"thread_start_us":622,"threads_started":1}
I20260806 02:50:37.725989 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66834c6ca26b45ad891689c87807227d" candidate_uuid: "7ec4af4579204039a1e4c072c0435792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb0691f7254e43ee833bba5171b544ee" is_pre_election: true
I20260806 02:50:37.726462 20739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66834c6ca26b45ad891689c87807227d" candidate_uuid: "7ec4af4579204039a1e4c072c0435792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce770dd6e4d7499aa8a57fbca40d2579" is_pre_election: true
I20260806 02:50:37.726725 20602 raft_consensus.cc:2454] T 66834c6ca26b45ad891689c87807227d P cb0691f7254e43ee833bba5171b544ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ec4af4579204039a1e4c072c0435792 in term 0.
I20260806 02:50:37.727078 20739 raft_consensus.cc:2454] T 66834c6ca26b45ad891689c87807227d P ce770dd6e4d7499aa8a57fbca40d2579 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ec4af4579204039a1e4c072c0435792 in term 0.
I20260806 02:50:37.727586 20399 leader_election.cc:302] T 66834c6ca26b45ad891689c87807227d P 7ec4af4579204039a1e4c072c0435792 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ec4af4579204039a1e4c072c0435792, cb0691f7254e43ee833bba5171b544ee; no voters:
I20260806 02:50:37.728114 20811 raft_consensus.cc:2781] T 66834c6ca26b45ad891689c87807227d P 7ec4af4579204039a1e4c072c0435792 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:50:37.728291 20811 raft_consensus.cc:482] T 66834c6ca26b45ad891689c87807227d P 7ec4af4579204039a1e4c072c0435792 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:50:37.728394 20811 raft_consensus.cc:3037] T 66834c6ca26b45ad891689c87807227d P 7ec4af4579204039a1e4c072c0435792 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:37.730799 20811 raft_consensus.cc:504] T 66834c6ca26b45ad891689c87807227d P 7ec4af4579204039a1e4c072c0435792 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec4af4579204039a1e4c072c0435792" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 37619 } } peers { permanent_uuid: "ce770dd6e4d7499aa8a57fbca40d2579" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45775 } } peers { permanent_uuid: "cb0691f7254e43ee833bba5171b544ee" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37131 } }
I20260806 02:50:37.731487 20811 leader_election.cc:288] T 66834c6ca26b45ad891689c87807227d P 7ec4af4579204039a1e4c072c0435792 [CANDIDATE]: Term 1 election: Requested vote from peers ce770dd6e4d7499aa8a57fbca40d2579 (127.1.210.67:45775), cb0691f7254e43ee833bba5171b544ee (127.1.210.66:37131)
I20260806 02:50:37.731904 20739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66834c6ca26b45ad891689c87807227d" candidate_uuid: "7ec4af4579204039a1e4c072c0435792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce770dd6e4d7499aa8a57fbca40d2579"
I20260806 02:50:37.732075 20602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66834c6ca26b45ad891689c87807227d" candidate_uuid: "7ec4af4579204039a1e4c072c0435792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb0691f7254e43ee833bba5171b544ee"
I20260806 02:50:37.732116 20739 raft_consensus.cc:3037] T 66834c6ca26b45ad891689c87807227d P ce770dd6e4d7499aa8a57fbca40d2579 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:37.732311 20602 raft_consensus.cc:3037] T 66834c6ca26b45ad891689c87807227d P cb0691f7254e43ee833bba5171b544ee [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:37.734462 20739 raft_consensus.cc:2454] T 66834c6ca26b45ad891689c87807227d P ce770dd6e4d7499aa8a57fbca40d2579 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ec4af4579204039a1e4c072c0435792 in term 1.
I20260806 02:50:37.734470 20602 raft_consensus.cc:2454] T 66834c6ca26b45ad891689c87807227d P cb0691f7254e43ee833bba5171b544ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ec4af4579204039a1e4c072c0435792 in term 1.
I20260806 02:50:37.734952 20401 leader_election.cc:302] T 66834c6ca26b45ad891689c87807227d P 7ec4af4579204039a1e4c072c0435792 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ec4af4579204039a1e4c072c0435792, ce770dd6e4d7499aa8a57fbca40d2579; no voters:
I20260806 02:50:37.735332 20811 raft_consensus.cc:2781] T 66834c6ca26b45ad891689c87807227d P 7ec4af4579204039a1e4c072c0435792 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:37.735847 20811 raft_consensus.cc:686] T 66834c6ca26b45ad891689c87807227d P 7ec4af4579204039a1e4c072c0435792 [term 1 LEADER]: Becoming Leader. State: Replica: 7ec4af4579204039a1e4c072c0435792, State: Running, Role: LEADER
I20260806 02:50:37.736348 20811 consensus_queue.cc:237] T 66834c6ca26b45ad891689c87807227d P 7ec4af4579204039a1e4c072c0435792 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec4af4579204039a1e4c072c0435792" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 37619 } } peers { permanent_uuid: "ce770dd6e4d7499aa8a57fbca40d2579" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45775 } } peers { permanent_uuid: "cb0691f7254e43ee833bba5171b544ee" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37131 } }
W20260806 02:50:37.738919 20787 tablet.cc:2369] T 66834c6ca26b45ad891689c87807227d P ce770dd6e4d7499aa8a57fbca40d2579: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:50:37.741520 20319 catalog_manager.cc:5495] T 66834c6ca26b45ad891689c87807227d P 7ec4af4579204039a1e4c072c0435792 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ec4af4579204039a1e4c072c0435792 (127.1.210.65). New cstate: current_term: 1 leader_uuid: "7ec4af4579204039a1e4c072c0435792" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec4af4579204039a1e4c072c0435792" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 37619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce770dd6e4d7499aa8a57fbca40d2579" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb0691f7254e43ee833bba5171b544ee" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37131 } health_report { overall_health: UNKNOWN } } }
I20260806 02:50:37.779857 20799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:37.774097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41820 (local address 127.1.210.65:37619)
0806 02:50:37.774332 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:37.774338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:37.774379 (+ 41us) server_negotiation.cc:407] Connection header received
0806 02:50:37.774495 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:37.774499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:37.774585 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:37.774800 (+ 215us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:50:37.775259 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.775997 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.777530 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.777955 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.778094 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:37.778708 (+ 614us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:50:37.779214 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:50:37.779402 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:37.779559 (+ 157us) server_negotiation.cc:299] Negotiation successful
0806 02:50:37.779620 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 02:50:37.788738 20739 raft_consensus.cc:1261] T 66834c6ca26b45ad891689c87807227d P ce770dd6e4d7499aa8a57fbca40d2579 [term 1 FOLLOWER]: Refusing update from remote peer 7ec4af4579204039a1e4c072c0435792: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:50:37.788738 20602 raft_consensus.cc:1261] T 66834c6ca26b45ad891689c87807227d P cb0691f7254e43ee833bba5171b544ee [term 1 FOLLOWER]: Refusing update from remote peer 7ec4af4579204039a1e4c072c0435792: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:50:37.789983 20811 consensus_queue.cc:1035] T 66834c6ca26b45ad891689c87807227d P 7ec4af4579204039a1e4c072c0435792 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce770dd6e4d7499aa8a57fbca40d2579" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 45775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:50:37.790433 20816 consensus_queue.cc:1035] T 66834c6ca26b45ad891689c87807227d P 7ec4af4579204039a1e4c072c0435792 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb0691f7254e43ee833bba5171b544ee" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 37131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:50:37.801909 20819 mvcc.cc:205] Tried to move back new op lower bound from 7315393076371050496 to 7315393076172132352. Current Snapshot: MvccSnapshot[applied={T|T < 7315393076371050496}]
I20260806 02:50:37.818640 20320 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:50:37.818988 20320 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:50:37.823069 20320 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ee88f33cffcf409896239db4039a8301: Sending DeleteTablet for 3 replicas of tablet 66834c6ca26b45ad891689c87807227d
I20260806 02:50:37.824307 20445 tablet_service.cc:1514] Processing DeleteTablet for tablet 66834c6ca26b45ad891689c87807227d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:37 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:41812
I20260806 02:50:37.824361 20719 tablet_service.cc:1514] Processing DeleteTablet for tablet 66834c6ca26b45ad891689c87807227d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:37 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:37472
I20260806 02:50:37.824381 20581 tablet_service.cc:1514] Processing DeleteTablet for tablet 66834c6ca26b45ad891689c87807227d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:37 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:50846
I20260806 02:50:37.827862 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 20382
I20260806 02:50:37.835220 20831 tablet_replica.cc:330] T 66834c6ca26b45ad891689c87807227d P ce770dd6e4d7499aa8a57fbca40d2579: stopping tablet replica
I20260806 02:50:37.835794 20831 raft_consensus.cc:2229] T 66834c6ca26b45ad891689c87807227d P ce770dd6e4d7499aa8a57fbca40d2579 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:50:37.836462 20831 raft_consensus.cc:2258] T 66834c6ca26b45ad891689c87807227d P ce770dd6e4d7499aa8a57fbca40d2579 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:37.837383 20830 tablet_replica.cc:330] T 66834c6ca26b45ad891689c87807227d P cb0691f7254e43ee833bba5171b544ee: stopping tablet replica
I20260806 02:50:37.837460 20829 tablet_replica.cc:330] T 66834c6ca26b45ad891689c87807227d P 7ec4af4579204039a1e4c072c0435792: stopping tablet replica
I20260806 02:50:37.837823 20829 raft_consensus.cc:2229] T 66834c6ca26b45ad891689c87807227d P 7ec4af4579204039a1e4c072c0435792 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:50:37.837818 20830 raft_consensus.cc:2229] T 66834c6ca26b45ad891689c87807227d P cb0691f7254e43ee833bba5171b544ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:50:37.838301 20830 raft_consensus.cc:2258] T 66834c6ca26b45ad891689c87807227d P cb0691f7254e43ee833bba5171b544ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:37.838521 20829 raft_consensus.cc:2258] T 66834c6ca26b45ad891689c87807227d P 7ec4af4579204039a1e4c072c0435792 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:37.840679 20829 ts_tablet_manager.cc:1905] T 66834c6ca26b45ad891689c87807227d P 7ec4af4579204039a1e4c072c0435792: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:37.840672 20830 ts_tablet_manager.cc:1905] T 66834c6ca26b45ad891689c87807227d P cb0691f7254e43ee833bba5171b544ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:37.848126 20830 ts_tablet_manager.cc:1918] T 66834c6ca26b45ad891689c87807227d P cb0691f7254e43ee833bba5171b544ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:50:37.848332 20830 log.cc:1196] T 66834c6ca26b45ad891689c87807227d P cb0691f7254e43ee833bba5171b544ee: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-1/wal/wals/66834c6ca26b45ad891689c87807227d
I20260806 02:50:37.848328 20829 ts_tablet_manager.cc:1918] T 66834c6ca26b45ad891689c87807227d P 7ec4af4579204039a1e4c072c0435792: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:50:37.848502 20829 log.cc:1196] T 66834c6ca26b45ad891689c87807227d P 7ec4af4579204039a1e4c072c0435792: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-0/wal/wals/66834c6ca26b45ad891689c87807227d
I20260806 02:50:37.848928 20830 ts_tablet_manager.cc:1939] T 66834c6ca26b45ad891689c87807227d P cb0691f7254e43ee833bba5171b544ee: Deleting consensus metadata
I20260806 02:50:37.849825 20829 ts_tablet_manager.cc:1939] T 66834c6ca26b45ad891689c87807227d P 7ec4af4579204039a1e4c072c0435792: Deleting consensus metadata
I20260806 02:50:37.851066 20304 catalog_manager.cc:4841] TS cb0691f7254e43ee833bba5171b544ee (127.1.210.66:37131): tablet 66834c6ca26b45ad891689c87807227d (table test-table [id=5c34ca88fec04b408a27bd9e5416dfe2]) successfully deleted
I20260806 02:50:37.851679 20831 ts_tablet_manager.cc:1905] T 66834c6ca26b45ad891689c87807227d P ce770dd6e4d7499aa8a57fbca40d2579: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:37.852918 20307 catalog_manager.cc:4841] TS 7ec4af4579204039a1e4c072c0435792 (127.1.210.65:37619): tablet 66834c6ca26b45ad891689c87807227d (table test-table [id=5c34ca88fec04b408a27bd9e5416dfe2]) successfully deleted
Aug 06 02:50:37 dist-test-slave-8tdl krb5kdc[20274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.65@KRBTEST.COM
I20260806 02:50:37.858155 20831 ts_tablet_manager.cc:1918] T 66834c6ca26b45ad891689c87807227d P ce770dd6e4d7499aa8a57fbca40d2579: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:50:37.858330 20831 log.cc:1196] T 66834c6ca26b45ad891689c87807227d P ce770dd6e4d7499aa8a57fbca40d2579: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984304136196-1865-0/minicluster-data/ts-2/wal/wals/66834c6ca26b45ad891689c87807227d
I20260806 02:50:37.858937 20831 ts_tablet_manager.cc:1939] T 66834c6ca26b45ad891689c87807227d P ce770dd6e4d7499aa8a57fbca40d2579: Deleting consensus metadata
I20260806 02:50:37.860625 20307 catalog_manager.cc:4841] TS ce770dd6e4d7499aa8a57fbca40d2579 (127.1.210.67:45775): tablet 66834c6ca26b45ad891689c87807227d (table test-table [id=5c34ca88fec04b408a27bd9e5416dfe2]) successfully deleted
I20260806 02:50:37.866470 20799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:37.829077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41836 (local address 127.1.210.65:37619)
0806 02:50:37.829266 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:37.829272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:37.843568 (+ 14296us) server_negotiation.cc:407] Connection header received
0806 02:50:37.844083 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:37.844089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:37.844194 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:37.844321 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:37.846249 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.847573 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.851341 (+ 3768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.852291 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.852435 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:37.858965 (+ 6530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:37.858990 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:37.859019 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:37.859075 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:37.863246 (+ 4171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:37.864052 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:37.864062 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:37.864066 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:37.864271 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:37.864687 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:37.864693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:37.864696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:37.865194 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:37.865511 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:37.865795 (+ 284us) server_negotiation.cc:299] Negotiation successful
0806 02:50:37.866067 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:50:37.867398 20506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:50:37.972867 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 20382
I20260806 02:50:37.994797 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 20517
Aug 06 02:50:38 dist-test-slave-8tdl krb5kdc[20274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.66@KRBTEST.COM
I20260806 02:50:38.014293 20802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:37.995656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50860 (local address 127.1.210.66:37131)
0806 02:50:37.995855 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:37.995860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:37.995880 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:50:37.995975 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:37.995979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:37.996043 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:37.996160 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:37.997433 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:37.998207 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:37.999819 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:38.000307 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:38.000528 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:38.005865 (+ 5337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:38.005922 (+ 57us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:38.005946 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:38.006015 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:38.009766 (+ 3751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:38.011044 (+ 1278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:38.011085 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:38.011088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:38.011488 (+ 400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:38.012317 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:38.012325 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:38.012328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:38.012868 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:38.013304 (+ 436us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:38.013629 (+ 325us) server_negotiation.cc:299] Negotiation successful
0806 02:50:38.013897 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 02:50:38.015197 20643 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:50:38.108691 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 20517
I20260806 02:50:38.128315 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 20655
Aug 06 02:50:38 dist-test-slave-8tdl krb5kdc[20274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.67@KRBTEST.COM
I20260806 02:50:38.144762 20803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:38.129254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37488 (local address 127.1.210.67:45775)
0806 02:50:38.129443 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:38.129450 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:38.129527 (+ 77us) server_negotiation.cc:407] Connection header received
0806 02:50:38.129759 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:38.129764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:38.129845 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:38.129960 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:38.131472 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:38.132232 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:38.133611 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:38.134033 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:38.134188 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:38.138857 (+ 4669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:38.138873 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:38.138889 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:38.138934 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:38.141734 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:38.142485 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:38.142494 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:38.142496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:38.142633 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:38.143179 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:38.143186 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:38.143190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:38.143547 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:38.143877 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:38.144171 (+ 294us) server_negotiation.cc:299] Negotiation successful
0806 02:50:38.144401 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 02:50:38.145481 20781 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:50:38.238539 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 20655
I20260806 02:50:38.256412 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 20287
I20260806 02:50:38.257539 20350 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:50:38.373539 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 20287
2026-08-06T02:50:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4634 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-06T02:50:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:38Z Disabled control of system clock
I20260806 02:50:38.466728 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:40113
--webserver_interface=127.1.210.126
--webserver_port=0
--builtin_ntp_servers=127.1.210.84:43187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:40113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 02:50:38.846217 20840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:38.846554 20840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:38.846650 20840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:38.854453 20840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:38.854585 20840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:38.854631 20840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:38.854745 20840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:38.854815 20840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:38.865000 20840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:43187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:40113
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:40113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:50:38.866396 20840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:38.868309 20840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:38.881057 20845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:38.881148 20848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:38.881225 20846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:38.882131 20840 server_base.cc:1034] running on GCE node
I20260806 02:50:38.883112 20840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:38.884687 20840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:38.886047 20840 hybrid_clock.cc:648] HybridClock initialized: now 1785984638885928 us; error 57 us; skew 500 ppm
I20260806 02:50:38.886528 20840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:50:38.889441 20840 webserver.cc:460] Webserver started at http://127.1.210.126:39467/ using document root <none> and password file <none>
I20260806 02:50:38.890259 20840 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:38.890419 20840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:38.890841 20840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:38.894029 20840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "01b181183d704ed7938d4ed81863f7bf"
format_stamp: "Formatted at 2026-08-06 02:50:38 on dist-test-slave-8tdl"
server_key: "4d42a6cba8b97a28c9357ed75f1407f0"
server_key_iv: "0b1c621fa21eb538911b6703ccaee0fa"
server_key_version: "encryptionkey@0"
I20260806 02:50:38.895051 20840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "01b181183d704ed7938d4ed81863f7bf"
format_stamp: "Formatted at 2026-08-06 02:50:38 on dist-test-slave-8tdl"
server_key: "4d42a6cba8b97a28c9357ed75f1407f0"
server_key_iv: "0b1c621fa21eb538911b6703ccaee0fa"
server_key_version: "encryptionkey@0"
I20260806 02:50:38.900939 20840 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260806 02:50:38.904909 20854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:38.906525 20840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:50:38.906785 20840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "01b181183d704ed7938d4ed81863f7bf"
format_stamp: "Formatted at 2026-08-06 02:50:38 on dist-test-slave-8tdl"
server_key: "4d42a6cba8b97a28c9357ed75f1407f0"
server_key_iv: "0b1c621fa21eb538911b6703ccaee0fa"
server_key_version: "encryptionkey@0"
I20260806 02:50:38.907074 20840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:38.940133 20840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:38.941097 20840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:38.941416 20840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:38.961506 20840 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:40113
I20260806 02:50:38.961529 20906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:40113 every 8 connection(s)
I20260806 02:50:38.963145 20840 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:38.967414 20907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:38.968045 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 20840
I20260806 02:50:38.968246 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:50:38.968708 1865 external_mini_cluster.cc:1428] Setting key 67688ce182935002e31f54fd753e2dda
I20260806 02:50:38.982092 20907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01b181183d704ed7938d4ed81863f7bf: Bootstrap starting.
I20260806 02:50:38.983788 20911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:38.970868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60904 (local address 127.1.210.126:40113)
0806 02:50:38.971989 (+ 1121us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:38.972004 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:38.972055 (+ 51us) server_negotiation.cc:407] Connection header received
0806 02:50:38.973811 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:38.973836 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:38.974386 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:38.974863 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:38.975499 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:38.976830 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:38.978938 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:38.979699 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:38.979834 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:38.981176 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:38.981184 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:38.981199 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:38.981781 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:38.981862 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:38.981884 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:38.981972 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:38.982288 (+ 316us) server_negotiation.cc:299] Negotiation successful
0806 02:50:38.982753 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":430,"threads_started":1}
I20260806 02:50:38.985612 20907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01b181183d704ed7938d4ed81863f7bf: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:38.986775 20907 log.cc:824] T 00000000000000000000000000000000 P 01b181183d704ed7938d4ed81863f7bf: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:38.989638 20907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01b181183d704ed7938d4ed81863f7bf: No bootstrap required, opened a new log
I20260806 02:50:38.997267 20907 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01b181183d704ed7938d4ed81863f7bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b181183d704ed7938d4ed81863f7bf" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 40113 } }
I20260806 02:50:38.997735 20907 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01b181183d704ed7938d4ed81863f7bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:38.997862 20907 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01b181183d704ed7938d4ed81863f7bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01b181183d704ed7938d4ed81863f7bf, State: Initialized, Role: FOLLOWER
I20260806 02:50:38.998517 20907 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01b181183d704ed7938d4ed81863f7bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b181183d704ed7938d4ed81863f7bf" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 40113 } }
I20260806 02:50:38.998850 20907 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01b181183d704ed7938d4ed81863f7bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:50:38.999007 20907 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01b181183d704ed7938d4ed81863f7bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:50:38.999197 20907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01b181183d704ed7938d4ed81863f7bf [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:39.001595 20907 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01b181183d704ed7938d4ed81863f7bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b181183d704ed7938d4ed81863f7bf" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 40113 } }
I20260806 02:50:39.002204 20907 leader_election.cc:302] T 00000000000000000000000000000000 P 01b181183d704ed7938d4ed81863f7bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01b181183d704ed7938d4ed81863f7bf; no voters:
I20260806 02:50:39.003135 20907 leader_election.cc:288] T 00000000000000000000000000000000 P 01b181183d704ed7938d4ed81863f7bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:50:39.003297 20913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01b181183d704ed7938d4ed81863f7bf [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:39.004171 20913 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01b181183d704ed7938d4ed81863f7bf [term 1 LEADER]: Becoming Leader. State: Replica: 01b181183d704ed7938d4ed81863f7bf, State: Running, Role: LEADER
I20260806 02:50:39.004742 20913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01b181183d704ed7938d4ed81863f7bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b181183d704ed7938d4ed81863f7bf" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 40113 } }
I20260806 02:50:39.005553 20907 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01b181183d704ed7938d4ed81863f7bf [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:39.007822 20915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01b181183d704ed7938d4ed81863f7bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01b181183d704ed7938d4ed81863f7bf. Latest consensus state: current_term: 1 leader_uuid: "01b181183d704ed7938d4ed81863f7bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b181183d704ed7938d4ed81863f7bf" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 40113 } } }
I20260806 02:50:39.008267 20915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01b181183d704ed7938d4ed81863f7bf [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:39.009291 20914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01b181183d704ed7938d4ed81863f7bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01b181183d704ed7938d4ed81863f7bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b181183d704ed7938d4ed81863f7bf" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 40113 } } }
I20260806 02:50:39.009606 20914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01b181183d704ed7938d4ed81863f7bf [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:39.009774 20919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:50:39.017382 20919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:39.026381 20919 catalog_manager.cc:1347] Generated new cluster ID: 7819930fcd4042c9bf4e2743ff9113ed
I20260806 02:50:39.026554 20919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:39.040053 20919 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:50:39.042043 20919 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:39.053984 20919 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01b181183d704ed7938d4ed81863f7bf: Generated new TSK 0
I20260806 02:50:39.055286 20919 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:50:39.073026 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.65:0
--local_ip_for_outbound_sockets=127.1.210.65
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:40113
--builtin_ntp_servers=127.1.210.84:43187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 02:50:39.432331 20933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:39.432696 20933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:39.432864 20933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:39.440431 20933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:39.440594 20933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:39.440786 20933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.65
I20260806 02:50:39.451136 20933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:43187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:40113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.65
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:50:39.452518 20933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:39.454394 20933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:39.468446 20938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:39.468504 20941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:39.468422 20939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:39.469305 20933 server_base.cc:1034] running on GCE node
I20260806 02:50:39.469952 20933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:39.471105 20933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:39.472352 20933 hybrid_clock.cc:648] HybridClock initialized: now 1785984639472261 us; error 75 us; skew 500 ppm
I20260806 02:50:39.472756 20933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:50:39.475625 20933 webserver.cc:460] Webserver started at http://127.1.210.65:38649/ using document root <none> and password file <none>
I20260806 02:50:39.476440 20933 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:39.476540 20933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:39.476927 20933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:39.480129 20933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-0/data/instance:
uuid: "0dc2102de5404edf9cf061b729164162"
format_stamp: "Formatted at 2026-08-06 02:50:39 on dist-test-slave-8tdl"
server_key: "4168f6057611db7629759982f53b72f5"
server_key_iv: "08685b80805e2b8cb44810dc115c7e2e"
server_key_version: "encryptionkey@0"
I20260806 02:50:39.480988 20933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance:
uuid: "0dc2102de5404edf9cf061b729164162"
format_stamp: "Formatted at 2026-08-06 02:50:39 on dist-test-slave-8tdl"
server_key: "4168f6057611db7629759982f53b72f5"
server_key_iv: "08685b80805e2b8cb44810dc115c7e2e"
server_key_version: "encryptionkey@0"
I20260806 02:50:39.486845 20933 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 02:50:39.490872 20948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:39.492260 20933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:50:39.492435 20933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-0/wal
uuid: "0dc2102de5404edf9cf061b729164162"
format_stamp: "Formatted at 2026-08-06 02:50:39 on dist-test-slave-8tdl"
server_key: "4168f6057611db7629759982f53b72f5"
server_key_iv: "08685b80805e2b8cb44810dc115c7e2e"
server_key_version: "encryptionkey@0"
I20260806 02:50:39.492671 20933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:39.513285 20933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:39.514228 20933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:39.514571 20933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:39.515789 20933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:39.517390 20933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:39.517480 20933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:39.517652 20933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:39.517758 20933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:39.553467 20933 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.65:40697
I20260806 02:50:39.553501 21062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.65:40697 every 8 connection(s)
I20260806 02:50:39.555054 20933 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
I20260806 02:50:39.556988 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 20933
I20260806 02:50:39.557193 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance
I20260806 02:50:39.557730 1865 external_mini_cluster.cc:1428] Setting key 6b42dc2f5c3bf15c035fb3a8df1158df
I20260806 02:50:39.560978 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.66:0
--local_ip_for_outbound_sockets=127.1.210.66
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:40113
--builtin_ntp_servers=127.1.210.84:43187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 02:50:39.569695 21065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:39.558385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:37545 (local address 127.1.210.126:40113)
0806 02:50:39.558942 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:39.558948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:39.560395 (+ 1447us) server_negotiation.cc:407] Connection header received
0806 02:50:39.562381 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:39.562387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:39.562487 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:39.562606 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:39.563754 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:39.564590 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:39.566642 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:39.567119 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:39.567257 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:39.568770 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:39.568774 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:39.568778 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:39.568798 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:39.568843 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:39.568855 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:39.568963 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:39.569197 (+ 234us) server_negotiation.cc:299] Negotiation successful
0806 02:50:39.569308 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":309,"threads_started":1}
I20260806 02:50:39.570837 21066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:39.559108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:40113 (local address 127.1.210.65:37545)
0806 02:50:39.560243 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:39.560286 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:39.562101 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:39.562877 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:39.562885 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:39.562979 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:39.563583 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:39.563600 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:39.564766 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:39.564770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:39.566469 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:39.566484 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:39.567381 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:39.567390 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:39.567649 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:39.568418 (+ 769us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:50:39.568421 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:39.568464 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:50:39.568554 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:39.568962 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:39.568977 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:39.568987 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 02:50:39.569182 (+ 195us) client_negotiation.cc:240] Negotiation successful
0806 02:50:39.569622 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":360,"threads_started":1}
I20260806 02:50:39.572525 21063 heartbeater.cc:347] Connected to a master server at 127.1.210.126:40113
I20260806 02:50:39.572902 21063 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:39.573678 21063 heartbeater.cc:511] Master 127.1.210.126:40113 requested a full tablet report, sending...
I20260806 02:50:39.575892 20872 ts_manager.cc:194] Registered new tserver with Master: 0dc2102de5404edf9cf061b729164162 (127.1.210.65:40697)
I20260806 02:50:39.577461 20872 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.210.65:37545
W20260806 02:50:39.924372 21067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:39.924726 21067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:39.924891 21067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:39.932492 21067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:39.932655 21067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:39.932832 21067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.66
I20260806 02:50:39.944032 21067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:43187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:40113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.66
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:50:39.945436 21067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:39.947532 21067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:39.962107 21072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:39.962232 21073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:39.962181 21075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:39.968942 21067 server_base.cc:1034] running on GCE node
I20260806 02:50:39.969753 21067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:39.971038 21067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:39.972321 21067 hybrid_clock.cc:648] HybridClock initialized: now 1785984639972245 us; error 72 us; skew 500 ppm
I20260806 02:50:39.972738 21067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:50:39.975965 21067 webserver.cc:460] Webserver started at http://127.1.210.66:41921/ using document root <none> and password file <none>
I20260806 02:50:39.976938 21067 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:39.977067 21067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:39.977416 21067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:39.981340 21067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-1/data/instance:
uuid: "03a18a5ada5645d3840f0f8ea013b5da"
format_stamp: "Formatted at 2026-08-06 02:50:39 on dist-test-slave-8tdl"
server_key: "61a1f501b22045b413b0f8ab446b869f"
server_key_iv: "e105837d0f323c4e68b3b78116aa3351"
server_key_version: "encryptionkey@0"
I20260806 02:50:39.982206 21067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance:
uuid: "03a18a5ada5645d3840f0f8ea013b5da"
format_stamp: "Formatted at 2026-08-06 02:50:39 on dist-test-slave-8tdl"
server_key: "61a1f501b22045b413b0f8ab446b869f"
server_key_iv: "e105837d0f323c4e68b3b78116aa3351"
server_key_version: "encryptionkey@0"
I20260806 02:50:39.988420 21067 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260806 02:50:39.992705 21081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:39.994448 21067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:50:39.994681 21067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-1/wal
uuid: "03a18a5ada5645d3840f0f8ea013b5da"
format_stamp: "Formatted at 2026-08-06 02:50:39 on dist-test-slave-8tdl"
server_key: "61a1f501b22045b413b0f8ab446b869f"
server_key_iv: "e105837d0f323c4e68b3b78116aa3351"
server_key_version: "encryptionkey@0"
I20260806 02:50:39.995038 21067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:40.008822 21067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:40.009967 21067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:40.010351 21067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:40.011703 21067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:40.013432 21067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:40.013525 21067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:40.013698 21067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:40.013782 21067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:40.049984 21067 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.66:46611
I20260806 02:50:40.050042 21195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.66:46611 every 8 connection(s)
I20260806 02:50:40.051508 21067 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
I20260806 02:50:40.055572 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 21067
I20260806 02:50:40.055809 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance
I20260806 02:50:40.056344 1865 external_mini_cluster.cc:1428] Setting key 4b8bdf2b980a6f9e399ad2816e41acb5
I20260806 02:50:40.059533 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.67:0
--local_ip_for_outbound_sockets=127.1.210.67
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:40113
--builtin_ntp_servers=127.1.210.84:43187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 02:50:40.065654 21065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:40.054146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:46933 (local address 127.1.210.126:40113)
0806 02:50:40.054401 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:40.054406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:40.056069 (+ 1663us) server_negotiation.cc:407] Connection header received
0806 02:50:40.058187 (+ 2118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:40.058193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:40.058296 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:40.058410 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:40.059662 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:40.060402 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:40.062170 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:40.062617 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:40.062851 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:40.064744 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:40.064749 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:40.064753 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:40.064775 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:40.064811 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:40.064824 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:40.064913 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:40.065264 (+ 351us) server_negotiation.cc:299] Negotiation successful
0806 02:50:40.065371 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 02:50:40.067297 21198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:40.054768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:40113 (local address 127.1.210.66:46933)
0806 02:50:40.055880 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:40.055940 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:40.057888 (+ 1948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:40.058718 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:40.058735 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:40.058851 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:40.059477 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:40.059495 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:40.060567 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:40.060586 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:40.061925 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:40.061939 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:40.062961 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:40.062971 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:40.063376 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:40.064269 (+ 893us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:50:40.064275 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:40.064347 (+ 72us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:50:40.064470 (+ 123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:40.064940 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:40.064955 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:40.064966 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 02:50:40.065196 (+ 230us) client_negotiation.cc:240] Negotiation successful
0806 02:50:40.065689 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":344,"threads_started":1}
I20260806 02:50:40.069031 21196 heartbeater.cc:347] Connected to a master server at 127.1.210.126:40113
I20260806 02:50:40.069381 21196 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:40.070209 21196 heartbeater.cc:511] Master 127.1.210.126:40113 requested a full tablet report, sending...
I20260806 02:50:40.071982 20872 ts_manager.cc:194] Registered new tserver with Master: 03a18a5ada5645d3840f0f8ea013b5da (127.1.210.66:46611)
I20260806 02:50:40.073038 20872 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.210.66:46933
W20260806 02:50:40.408941 21199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:40.409252 21199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:40.409343 21199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:40.417179 21199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:40.417294 21199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:40.417403 21199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.67
I20260806 02:50:40.426884 21199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:43187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:40113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.67
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:50:40.428264 21199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:40.430105 21199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:40.443887 21205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:40.443918 21207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:40.443982 21204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:40.445230 21199 server_base.cc:1034] running on GCE node
I20260806 02:50:40.445914 21199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:40.447005 21199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:40.448254 21199 hybrid_clock.cc:648] HybridClock initialized: now 1785984640448190 us; error 59 us; skew 500 ppm
I20260806 02:50:40.448648 21199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:50:40.451297 21199 webserver.cc:460] Webserver started at http://127.1.210.67:42359/ using document root <none> and password file <none>
I20260806 02:50:40.452047 21199 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:40.452140 21199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:40.452519 21199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:40.455631 21199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-2/data/instance:
uuid: "476b7d1d0f4f46f2b896125fccfd3ec1"
format_stamp: "Formatted at 2026-08-06 02:50:40 on dist-test-slave-8tdl"
server_key: "2e08f809be3aed992e08310bd6ffefe4"
server_key_iv: "6f12e2365b38f99547ca6a03fa4b3a46"
server_key_version: "encryptionkey@0"
I20260806 02:50:40.456451 21199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance:
uuid: "476b7d1d0f4f46f2b896125fccfd3ec1"
format_stamp: "Formatted at 2026-08-06 02:50:40 on dist-test-slave-8tdl"
server_key: "2e08f809be3aed992e08310bd6ffefe4"
server_key_iv: "6f12e2365b38f99547ca6a03fa4b3a46"
server_key_version: "encryptionkey@0"
I20260806 02:50:40.461699 21199 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260806 02:50:40.465525 21213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:40.467196 21199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 02:50:40.467367 21199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-2/wal
uuid: "476b7d1d0f4f46f2b896125fccfd3ec1"
format_stamp: "Formatted at 2026-08-06 02:50:40 on dist-test-slave-8tdl"
server_key: "2e08f809be3aed992e08310bd6ffefe4"
server_key_iv: "6f12e2365b38f99547ca6a03fa4b3a46"
server_key_version: "encryptionkey@0"
I20260806 02:50:40.467589 21199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:40.497880 21199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:40.498911 21199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:40.499251 21199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:40.500524 21199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:40.502044 21199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:40.502131 21199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:40.502301 21199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:40.502383 21199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:40.537719 21199 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.67:44301
I20260806 02:50:40.537812 21326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.67:44301 every 8 connection(s)
I20260806 02:50:40.539203 21199 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
I20260806 02:50:40.544742 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 21199
I20260806 02:50:40.544948 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance
I20260806 02:50:40.545444 1865 external_mini_cluster.cc:1428] Setting key 0422d2239410c7b304221b21fcd5c5ce
I20260806 02:50:40.552603 21065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:40.541909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:42807 (local address 127.1.210.126:40113)
0806 02:50:40.542107 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:40.542113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:40.543626 (+ 1513us) server_negotiation.cc:407] Connection header received
0806 02:50:40.545640 (+ 2014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:40.545646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:40.545747 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:40.545863 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:40.547083 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:40.547905 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:40.549502 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:40.549936 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:40.550083 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:40.551616 (+ 1533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:40.551621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:40.551624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:40.551644 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:40.551688 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:40.551701 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:40.551803 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:40.552116 (+ 313us) server_negotiation.cc:299] Negotiation successful
0806 02:50:40.552223 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 02:50:40.553706 21329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:40.542455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:40113 (local address 127.1.210.67:42807)
0806 02:50:40.543446 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:40.543486 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:40.545332 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:40.546121 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:40.546131 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:40.546225 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:40.546899 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:40.546919 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:40.548078 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:40.548082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:40.549349 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:40.549361 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:40.550189 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:40.550196 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:40.550447 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:40.551252 (+ 805us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:50:40.551255 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:40.551302 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:50:40.551383 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:40.551802 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:40.551814 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:40.551824 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 02:50:40.552097 (+ 273us) client_negotiation.cc:240] Negotiation successful
0806 02:50:40.552520 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":326,"threads_started":1}
I20260806 02:50:40.555210 21327 heartbeater.cc:347] Connected to a master server at 127.1.210.126:40113
I20260806 02:50:40.555576 21327 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:40.556605 21327 heartbeater.cc:511] Master 127.1.210.126:40113 requested a full tablet report, sending...
I20260806 02:50:40.558431 20872 ts_manager.cc:194] Registered new tserver with Master: 476b7d1d0f4f46f2b896125fccfd3ec1 (127.1.210.67:44301)
I20260806 02:50:40.559541 20872 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.210.67:42807
I20260806 02:50:40.561245 1865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:50:40.572618 21065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:40.565682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60910 (local address 127.1.210.126:40113)
0806 02:50:40.565898 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:40.565903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:40.566446 (+ 543us) server_negotiation.cc:407] Connection header received
0806 02:50:40.566595 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:40.566600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:40.566731 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:40.566839 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:40.567479 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:40.568291 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:40.569752 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:40.570214 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:40.570355 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:40.571712 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:40.571717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:40.571721 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:40.571740 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:40.571773 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:40.571805 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:40.571972 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:40.572199 (+ 227us) server_negotiation.cc:299] Negotiation successful
0806 02:50:40.572263 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 02:50:40.580654 21063 heartbeater.cc:503] Master 127.1.210.126:40113 was elected leader, sending a full tablet report...
I20260806 02:50:40.577476 20872 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60910:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:50:40.585304 20872 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:50:40.620579 21341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:40.598793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:46611 (local address 127.0.0.1:51916)
0806 02:50:40.609906 (+ 11113us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:40.609983 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:40.610258 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:40.612587 (+ 2329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:40.612596 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:40.612660 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:40.613310 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:40.613329 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:40.615361 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:40.615367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:40.617606 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:40.617622 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:40.619473 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:40.619479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:40.619671 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:40.619728 (+ 57us) client_negotiation.cc:769] Sending connection context
0806 02:50:40.619896 (+ 168us) client_negotiation.cc:240] Negotiation successful
0806 02:50:40.620116 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10702,"mutex_wait_us":227,"spinlock_wait_cycles":7296,"thread_start_us":12204,"threads_started":1}
I20260806 02:50:40.620877 21340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:40.599004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51916 (local address 127.1.210.66:46611)
0806 02:50:40.611688 (+ 12684us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:40.611697 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:40.611726 (+ 29us) server_negotiation.cc:407] Connection header received
0806 02:50:40.611821 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:40.611827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:40.612038 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:40.612263 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:40.613562 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:40.615184 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:40.617857 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:40.619235 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:40.619391 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:40.619664 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:40.620281 (+ 617us) server_negotiation.cc:299] Negotiation successful
0806 02:50:40.620495 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12532,"thread_start_us":12352,"threads_started":1}
I20260806 02:50:40.621194 21342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:40.599256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43270 (local address 127.1.210.67:44301)
0806 02:50:40.611032 (+ 11776us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:40.611046 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:40.611428 (+ 382us) server_negotiation.cc:407] Connection header received
0806 02:50:40.611732 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:40.611742 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:40.612044 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:40.612262 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:40.613874 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:40.615529 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:40.618235 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:40.619724 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:40.619865 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:40.620134 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:40.620676 (+ 542us) server_negotiation.cc:299] Negotiation successful
0806 02:50:40.620848 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11524,"thread_start_us":11275,"threads_started":1}
I20260806 02:50:40.621809 21339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:40.599146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:44301 (local address 127.0.0.1:43270)
0806 02:50:40.611311 (+ 12165us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:40.611330 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:40.611546 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:40.612587 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:40.612596 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:40.612659 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:40.613672 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:40.613683 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:40.615677 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:40.615687 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:40.617968 (+ 2281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:40.617984 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:40.619950 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:40.619956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:40.620228 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:40.620295 (+ 67us) client_negotiation.cc:769] Sending connection context
0806 02:50:40.620393 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 02:50:40.620532 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12026,"mutex_wait_us":33,"spinlock_wait_cycles":3200,"thread_start_us":10177,"threads_started":1}
I20260806 02:50:40.623317 21344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:40.612113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:40697 (local address 127.0.0.1:33228)
0806 02:50:40.614250 (+ 2137us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:40.614269 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:40.614395 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:40.615373 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:40.615377 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:40.615408 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:40.615883 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:40.615917 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:40.618004 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:40.618011 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:40.620280 (+ 2269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:40.620296 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:40.622470 (+ 2174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:40.622477 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:40.622711 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:40.622745 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 02:50:40.622839 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 02:50:40.622976 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1971,"mutex_wait_us":91,"thread_start_us":1817,"threads_started":1}
I20260806 02:50:40.624155 21343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:40.611884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33228 (local address 127.1.210.65:40697)
0806 02:50:40.613468 (+ 1584us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:40.613479 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:40.614530 (+ 1051us) server_negotiation.cc:407] Connection header received
0806 02:50:40.614644 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:40.614652 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:40.614951 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:40.615164 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:40.616092 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:40.617818 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:40.620520 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:40.621942 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:40.622074 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:40.622337 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:40.623581 (+ 1244us) server_negotiation.cc:299] Negotiation successful
0806 02:50:40.623781 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1398,"thread_start_us":1228,"threads_started":1}
I20260806 02:50:40.625640 21261 tablet_service.cc:1467] Processing CreateTablet for tablet cae06795f2824ec999a58e7357dbaf92 (DEFAULT_TABLE table=test-table [id=a36a4057de9f42f8b48fd3f12a84ca2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:40.625713 21129 tablet_service.cc:1467] Processing CreateTablet for tablet cae06795f2824ec999a58e7357dbaf92 (DEFAULT_TABLE table=test-table [id=a36a4057de9f42f8b48fd3f12a84ca2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:40.627239 21129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cae06795f2824ec999a58e7357dbaf92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:40.627542 21261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cae06795f2824ec999a58e7357dbaf92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:40.628033 20996 tablet_service.cc:1467] Processing CreateTablet for tablet cae06795f2824ec999a58e7357dbaf92 (DEFAULT_TABLE table=test-table [id=a36a4057de9f42f8b48fd3f12a84ca2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:40.629441 20996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cae06795f2824ec999a58e7357dbaf92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:40.640229 21345 tablet_bootstrap.cc:492] T cae06795f2824ec999a58e7357dbaf92 P 476b7d1d0f4f46f2b896125fccfd3ec1: Bootstrap starting.
I20260806 02:50:40.643481 21346 tablet_bootstrap.cc:492] T cae06795f2824ec999a58e7357dbaf92 P 03a18a5ada5645d3840f0f8ea013b5da: Bootstrap starting.
I20260806 02:50:40.645699 21345 tablet_bootstrap.cc:654] T cae06795f2824ec999a58e7357dbaf92 P 476b7d1d0f4f46f2b896125fccfd3ec1: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:40.647033 21347 tablet_bootstrap.cc:492] T cae06795f2824ec999a58e7357dbaf92 P 0dc2102de5404edf9cf061b729164162: Bootstrap starting.
I20260806 02:50:40.647329 21346 tablet_bootstrap.cc:654] T cae06795f2824ec999a58e7357dbaf92 P 03a18a5ada5645d3840f0f8ea013b5da: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:40.647375 21345 log.cc:824] T cae06795f2824ec999a58e7357dbaf92 P 476b7d1d0f4f46f2b896125fccfd3ec1: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:40.648630 21346 log.cc:824] T cae06795f2824ec999a58e7357dbaf92 P 03a18a5ada5645d3840f0f8ea013b5da: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:40.650810 21345 tablet_bootstrap.cc:492] T cae06795f2824ec999a58e7357dbaf92 P 476b7d1d0f4f46f2b896125fccfd3ec1: No bootstrap required, opened a new log
I20260806 02:50:40.650849 21347 tablet_bootstrap.cc:654] T cae06795f2824ec999a58e7357dbaf92 P 0dc2102de5404edf9cf061b729164162: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:40.651134 21345 ts_tablet_manager.cc:1397] T cae06795f2824ec999a58e7357dbaf92 P 476b7d1d0f4f46f2b896125fccfd3ec1: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260806 02:50:40.651873 21346 tablet_bootstrap.cc:492] T cae06795f2824ec999a58e7357dbaf92 P 03a18a5ada5645d3840f0f8ea013b5da: No bootstrap required, opened a new log
I20260806 02:50:40.651978 21347 log.cc:824] T cae06795f2824ec999a58e7357dbaf92 P 0dc2102de5404edf9cf061b729164162: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:40.652145 21346 ts_tablet_manager.cc:1397] T cae06795f2824ec999a58e7357dbaf92 P 03a18a5ada5645d3840f0f8ea013b5da: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260806 02:50:40.655575 21347 tablet_bootstrap.cc:492] T cae06795f2824ec999a58e7357dbaf92 P 0dc2102de5404edf9cf061b729164162: No bootstrap required, opened a new log
I20260806 02:50:40.655905 21347 ts_tablet_manager.cc:1397] T cae06795f2824ec999a58e7357dbaf92 P 0dc2102de5404edf9cf061b729164162: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260806 02:50:40.662256 21345 raft_consensus.cc:348] T cae06795f2824ec999a58e7357dbaf92 P 476b7d1d0f4f46f2b896125fccfd3ec1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03a18a5ada5645d3840f0f8ea013b5da" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 46611 } } peers { permanent_uuid: "476b7d1d0f4f46f2b896125fccfd3ec1" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44301 } } peers { permanent_uuid: "0dc2102de5404edf9cf061b729164162" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 40697 } }
I20260806 02:50:40.662861 21345 raft_consensus.cc:374] T cae06795f2824ec999a58e7357dbaf92 P 476b7d1d0f4f46f2b896125fccfd3ec1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:40.663029 21345 raft_consensus.cc:729] T cae06795f2824ec999a58e7357dbaf92 P 476b7d1d0f4f46f2b896125fccfd3ec1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 476b7d1d0f4f46f2b896125fccfd3ec1, State: Initialized, Role: FOLLOWER
I20260806 02:50:40.662797 21346 raft_consensus.cc:348] T cae06795f2824ec999a58e7357dbaf92 P 03a18a5ada5645d3840f0f8ea013b5da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03a18a5ada5645d3840f0f8ea013b5da" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 46611 } } peers { permanent_uuid: "476b7d1d0f4f46f2b896125fccfd3ec1" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44301 } } peers { permanent_uuid: "0dc2102de5404edf9cf061b729164162" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 40697 } }
I20260806 02:50:40.663316 21346 raft_consensus.cc:374] T cae06795f2824ec999a58e7357dbaf92 P 03a18a5ada5645d3840f0f8ea013b5da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:40.663035 21347 raft_consensus.cc:348] T cae06795f2824ec999a58e7357dbaf92 P 0dc2102de5404edf9cf061b729164162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03a18a5ada5645d3840f0f8ea013b5da" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 46611 } } peers { permanent_uuid: "476b7d1d0f4f46f2b896125fccfd3ec1" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44301 } } peers { permanent_uuid: "0dc2102de5404edf9cf061b729164162" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 40697 } }
I20260806 02:50:40.663486 21346 raft_consensus.cc:729] T cae06795f2824ec999a58e7357dbaf92 P 03a18a5ada5645d3840f0f8ea013b5da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03a18a5ada5645d3840f0f8ea013b5da, State: Initialized, Role: FOLLOWER
I20260806 02:50:40.663620 21347 raft_consensus.cc:374] T cae06795f2824ec999a58e7357dbaf92 P 0dc2102de5404edf9cf061b729164162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:40.663790 21347 raft_consensus.cc:729] T cae06795f2824ec999a58e7357dbaf92 P 0dc2102de5404edf9cf061b729164162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dc2102de5404edf9cf061b729164162, State: Initialized, Role: FOLLOWER
I20260806 02:50:40.663736 21345 consensus_queue.cc:260] T cae06795f2824ec999a58e7357dbaf92 P 476b7d1d0f4f46f2b896125fccfd3ec1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03a18a5ada5645d3840f0f8ea013b5da" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 46611 } } peers { permanent_uuid: "476b7d1d0f4f46f2b896125fccfd3ec1" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44301 } } peers { permanent_uuid: "0dc2102de5404edf9cf061b729164162" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 40697 } }
I20260806 02:50:40.664098 21346 consensus_queue.cc:260] T cae06795f2824ec999a58e7357dbaf92 P 03a18a5ada5645d3840f0f8ea013b5da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03a18a5ada5645d3840f0f8ea013b5da" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 46611 } } peers { permanent_uuid: "476b7d1d0f4f46f2b896125fccfd3ec1" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44301 } } peers { permanent_uuid: "0dc2102de5404edf9cf061b729164162" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 40697 } }
I20260806 02:50:40.664885 21347 consensus_queue.cc:260] T cae06795f2824ec999a58e7357dbaf92 P 0dc2102de5404edf9cf061b729164162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03a18a5ada5645d3840f0f8ea013b5da" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 46611 } } peers { permanent_uuid: "476b7d1d0f4f46f2b896125fccfd3ec1" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44301 } } peers { permanent_uuid: "0dc2102de5404edf9cf061b729164162" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 40697 } }
I20260806 02:50:40.665038 21327 heartbeater.cc:503] Master 127.1.210.126:40113 was elected leader, sending a full tablet report...
I20260806 02:50:40.665750 21345 ts_tablet_manager.cc:1428] T cae06795f2824ec999a58e7357dbaf92 P 476b7d1d0f4f46f2b896125fccfd3ec1: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260806 02:50:40.667690 21196 heartbeater.cc:503] Master 127.1.210.126:40113 was elected leader, sending a full tablet report...
I20260806 02:50:40.668444 21346 ts_tablet_manager.cc:1428] T cae06795f2824ec999a58e7357dbaf92 P 03a18a5ada5645d3840f0f8ea013b5da: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260806 02:50:40.670145 21347 ts_tablet_manager.cc:1428] T cae06795f2824ec999a58e7357dbaf92 P 0dc2102de5404edf9cf061b729164162: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
W20260806 02:50:40.791079 21328 tablet.cc:2369] T cae06795f2824ec999a58e7357dbaf92 P 476b7d1d0f4f46f2b896125fccfd3ec1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:50:40.803716 21197 tablet.cc:2369] T cae06795f2824ec999a58e7357dbaf92 P 03a18a5ada5645d3840f0f8ea013b5da: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:50:40.807595 21064 tablet.cc:2369] T cae06795f2824ec999a58e7357dbaf92 P 0dc2102de5404edf9cf061b729164162: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:50:40.903461 21352 raft_consensus.cc:482] T cae06795f2824ec999a58e7357dbaf92 P 03a18a5ada5645d3840f0f8ea013b5da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:50:40.903823 21352 raft_consensus.cc:504] T cae06795f2824ec999a58e7357dbaf92 P 03a18a5ada5645d3840f0f8ea013b5da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03a18a5ada5645d3840f0f8ea013b5da" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 46611 } } peers { permanent_uuid: "476b7d1d0f4f46f2b896125fccfd3ec1" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44301 } } peers { permanent_uuid: "0dc2102de5404edf9cf061b729164162" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 40697 } }
I20260806 02:50:40.905645 21352 leader_election.cc:288] T cae06795f2824ec999a58e7357dbaf92 P 03a18a5ada5645d3840f0f8ea013b5da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 476b7d1d0f4f46f2b896125fccfd3ec1 (127.1.210.67:44301), 0dc2102de5404edf9cf061b729164162 (127.1.210.65:40697)
I20260806 02:50:40.915151 21343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:40.906211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:33973 (local address 127.1.210.65:40697)
0806 02:50:40.906422 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:40.906430 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:40.907259 (+ 829us) server_negotiation.cc:407] Connection header received
0806 02:50:40.907457 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:40.907462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:40.907552 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:40.907647 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:40.908544 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:40.909680 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:40.912926 (+ 3246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:40.914177 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:40.914329 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:40.914404 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:40.914741 (+ 337us) server_negotiation.cc:299] Negotiation successful
0806 02:50:40.914889 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 02:50:40.915182 21342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:40.906134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:54939 (local address 127.1.210.67:44301)
0806 02:50:40.906357 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:40.906363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:40.907255 (+ 892us) server_negotiation.cc:407] Connection header received
0806 02:50:40.907450 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:40.907455 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:40.907541 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:40.907647 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:40.908544 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:40.909680 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:40.912760 (+ 3080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:40.914063 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:40.914297 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:40.914375 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:40.914741 (+ 366us) server_negotiation.cc:299] Negotiation successful
0806 02:50:40.914889 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 02:50:40.915374 21354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:40.906174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:44301 (local address 127.1.210.66:54939)
0806 02:50:40.907094 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:40.907117 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:40.907298 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:40.907679 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:40.907684 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:40.907733 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:40.908384 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:40.908393 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:40.909841 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:40.909845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:40.912475 (+ 2630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:40.912491 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:40.914302 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:40.914308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:40.914513 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:40.914571 (+ 58us) client_negotiation.cc:769] Sending connection context
0806 02:50:40.914703 (+ 132us) client_negotiation.cc:240] Negotiation successful
0806 02:50:40.914957 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"mutex_wait_us":187,"spinlock_wait_cycles":96640,"thread_start_us":372,"threads_started":1}
I20260806 02:50:40.915863 21355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:40.906198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:40697 (local address 127.1.210.66:33973)
0806 02:50:40.907142 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:40.907163 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:40.907290 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:40.907679 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:40.907684 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:40.907731 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:40.908318 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:40.908337 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:40.909839 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:40.909845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:40.912733 (+ 2888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:40.912745 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:40.914358 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:40.914363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:40.914528 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:40.914582 (+ 54us) client_negotiation.cc:769] Sending connection context
0806 02:50:40.914693 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 02:50:40.914913 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"mutex_wait_us":610,"spinlock_wait_cycles":16384,"thread_start_us":510,"threads_started":1}
I20260806 02:50:40.916495 21281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cae06795f2824ec999a58e7357dbaf92" candidate_uuid: "03a18a5ada5645d3840f0f8ea013b5da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "476b7d1d0f4f46f2b896125fccfd3ec1" is_pre_election: true
I20260806 02:50:40.916994 21016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cae06795f2824ec999a58e7357dbaf92" candidate_uuid: "03a18a5ada5645d3840f0f8ea013b5da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc2102de5404edf9cf061b729164162" is_pre_election: true
I20260806 02:50:40.917176 21281 raft_consensus.cc:2454] T cae06795f2824ec999a58e7357dbaf92 P 476b7d1d0f4f46f2b896125fccfd3ec1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03a18a5ada5645d3840f0f8ea013b5da in term 0.
I20260806 02:50:40.917569 21016 raft_consensus.cc:2454] T cae06795f2824ec999a58e7357dbaf92 P 0dc2102de5404edf9cf061b729164162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03a18a5ada5645d3840f0f8ea013b5da in term 0.
I20260806 02:50:40.917949 21084 leader_election.cc:302] T cae06795f2824ec999a58e7357dbaf92 P 03a18a5ada5645d3840f0f8ea013b5da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03a18a5ada5645d3840f0f8ea013b5da, 476b7d1d0f4f46f2b896125fccfd3ec1; no voters:
I20260806 02:50:40.918378 21352 raft_consensus.cc:2781] T cae06795f2824ec999a58e7357dbaf92 P 03a18a5ada5645d3840f0f8ea013b5da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:50:40.918566 21352 raft_consensus.cc:482] T cae06795f2824ec999a58e7357dbaf92 P 03a18a5ada5645d3840f0f8ea013b5da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:50:40.918713 21352 raft_consensus.cc:3037] T cae06795f2824ec999a58e7357dbaf92 P 03a18a5ada5645d3840f0f8ea013b5da [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:40.921062 21352 raft_consensus.cc:504] T cae06795f2824ec999a58e7357dbaf92 P 03a18a5ada5645d3840f0f8ea013b5da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03a18a5ada5645d3840f0f8ea013b5da" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 46611 } } peers { permanent_uuid: "476b7d1d0f4f46f2b896125fccfd3ec1" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44301 } } peers { permanent_uuid: "0dc2102de5404edf9cf061b729164162" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 40697 } }
I20260806 02:50:40.921708 21352 leader_election.cc:288] T cae06795f2824ec999a58e7357dbaf92 P 03a18a5ada5645d3840f0f8ea013b5da [CANDIDATE]: Term 1 election: Requested vote from peers 476b7d1d0f4f46f2b896125fccfd3ec1 (127.1.210.67:44301), 0dc2102de5404edf9cf061b729164162 (127.1.210.65:40697)
I20260806 02:50:40.922200 21281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cae06795f2824ec999a58e7357dbaf92" candidate_uuid: "03a18a5ada5645d3840f0f8ea013b5da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "476b7d1d0f4f46f2b896125fccfd3ec1"
I20260806 02:50:40.922207 21016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cae06795f2824ec999a58e7357dbaf92" candidate_uuid: "03a18a5ada5645d3840f0f8ea013b5da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dc2102de5404edf9cf061b729164162"
I20260806 02:50:40.922417 21281 raft_consensus.cc:3037] T cae06795f2824ec999a58e7357dbaf92 P 476b7d1d0f4f46f2b896125fccfd3ec1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:40.922429 21016 raft_consensus.cc:3037] T cae06795f2824ec999a58e7357dbaf92 P 0dc2102de5404edf9cf061b729164162 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:40.924700 21281 raft_consensus.cc:2454] T cae06795f2824ec999a58e7357dbaf92 P 476b7d1d0f4f46f2b896125fccfd3ec1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03a18a5ada5645d3840f0f8ea013b5da in term 1.
I20260806 02:50:40.924768 21016 raft_consensus.cc:2454] T cae06795f2824ec999a58e7357dbaf92 P 0dc2102de5404edf9cf061b729164162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03a18a5ada5645d3840f0f8ea013b5da in term 1.
I20260806 02:50:40.925149 21084 leader_election.cc:302] T cae06795f2824ec999a58e7357dbaf92 P 03a18a5ada5645d3840f0f8ea013b5da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03a18a5ada5645d3840f0f8ea013b5da, 476b7d1d0f4f46f2b896125fccfd3ec1; no voters:
I20260806 02:50:40.925433 21352 raft_consensus.cc:2781] T cae06795f2824ec999a58e7357dbaf92 P 03a18a5ada5645d3840f0f8ea013b5da [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:40.925885 21352 raft_consensus.cc:686] T cae06795f2824ec999a58e7357dbaf92 P 03a18a5ada5645d3840f0f8ea013b5da [term 1 LEADER]: Becoming Leader. State: Replica: 03a18a5ada5645d3840f0f8ea013b5da, State: Running, Role: LEADER
I20260806 02:50:40.926388 21352 consensus_queue.cc:237] T cae06795f2824ec999a58e7357dbaf92 P 03a18a5ada5645d3840f0f8ea013b5da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03a18a5ada5645d3840f0f8ea013b5da" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 46611 } } peers { permanent_uuid: "476b7d1d0f4f46f2b896125fccfd3ec1" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44301 } } peers { permanent_uuid: "0dc2102de5404edf9cf061b729164162" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 40697 } }
I20260806 02:50:40.931784 20869 catalog_manager.cc:5495] T cae06795f2824ec999a58e7357dbaf92 P 03a18a5ada5645d3840f0f8ea013b5da reported cstate change: term changed from 0 to 1, leader changed from <none> to 03a18a5ada5645d3840f0f8ea013b5da (127.1.210.66). New cstate: current_term: 1 leader_uuid: "03a18a5ada5645d3840f0f8ea013b5da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03a18a5ada5645d3840f0f8ea013b5da" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 46611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476b7d1d0f4f46f2b896125fccfd3ec1" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dc2102de5404edf9cf061b729164162" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 40697 } health_report { overall_health: UNKNOWN } } }
I20260806 02:50:40.997112 21340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:40.991018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51918 (local address 127.1.210.66:46611)
0806 02:50:40.991305 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:40.991312 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:40.991421 (+ 109us) server_negotiation.cc:407] Connection header received
0806 02:50:40.991690 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:40.991694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:40.991797 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:40.991961 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:50:40.992368 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:40.993124 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:40.994550 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:40.995012 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:40.995156 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:40.995865 (+ 709us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:50:40.996443 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:50:40.996641 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:40.996761 (+ 120us) server_negotiation.cc:299] Negotiation successful
0806 02:50:40.996810 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 02:50:41.006137 21016 raft_consensus.cc:1261] T cae06795f2824ec999a58e7357dbaf92 P 0dc2102de5404edf9cf061b729164162 [term 1 FOLLOWER]: Refusing update from remote peer 03a18a5ada5645d3840f0f8ea013b5da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:50:41.006187 21281 raft_consensus.cc:1261] T cae06795f2824ec999a58e7357dbaf92 P 476b7d1d0f4f46f2b896125fccfd3ec1 [term 1 FOLLOWER]: Refusing update from remote peer 03a18a5ada5645d3840f0f8ea013b5da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:50:41.007359 21356 consensus_queue.cc:1035] T cae06795f2824ec999a58e7357dbaf92 P 03a18a5ada5645d3840f0f8ea013b5da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dc2102de5404edf9cf061b729164162" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:50:41.007784 21352 consensus_queue.cc:1035] T cae06795f2824ec999a58e7357dbaf92 P 03a18a5ada5645d3840f0f8ea013b5da [LEADER]: Connected to new peer: Peer: permanent_uuid: "476b7d1d0f4f46f2b896125fccfd3ec1" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 44301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:50:41.020591 21359 mvcc.cc:205] Tried to move back new op lower bound from 7315393089549996032 to 7315393089239150592. Current Snapshot: MvccSnapshot[applied={T|T < 7315393089549996032}]
I20260806 02:50:41.021494 21361 mvcc.cc:205] Tried to move back new op lower bound from 7315393089549996032 to 7315393089239150592. Current Snapshot: MvccSnapshot[applied={T|T < 7315393089549996032}]
I20260806 02:50:41.035615 20869 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:50:41.035943 20869 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:50:41.040309 20869 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 01b181183d704ed7938d4ed81863f7bf: Sending DeleteTablet for 3 replicas of tablet cae06795f2824ec999a58e7357dbaf92
I20260806 02:50:41.041674 21261 tablet_service.cc:1514] Processing DeleteTablet for tablet cae06795f2824ec999a58e7357dbaf92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:41 UTC) from {username='slave'} at 127.0.0.1:43270
I20260806 02:50:41.041832 20996 tablet_service.cc:1514] Processing DeleteTablet for tablet cae06795f2824ec999a58e7357dbaf92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:41 UTC) from {username='slave'} at 127.0.0.1:33228
I20260806 02:50:41.042521 21129 tablet_service.cc:1514] Processing DeleteTablet for tablet cae06795f2824ec999a58e7357dbaf92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:41 UTC) from {username='slave'} at 127.0.0.1:51916
I20260806 02:50:41.042667 21368 tablet_replica.cc:330] T cae06795f2824ec999a58e7357dbaf92 P 476b7d1d0f4f46f2b896125fccfd3ec1: stopping tablet replica
I20260806 02:50:41.043551 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 20933
I20260806 02:50:41.043699 21368 raft_consensus.cc:2229] T cae06795f2824ec999a58e7357dbaf92 P 476b7d1d0f4f46f2b896125fccfd3ec1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:50:41.044436 21368 raft_consensus.cc:2258] T cae06795f2824ec999a58e7357dbaf92 P 476b7d1d0f4f46f2b896125fccfd3ec1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:41.052525 21370 tablet_replica.cc:330] T cae06795f2824ec999a58e7357dbaf92 P 03a18a5ada5645d3840f0f8ea013b5da: stopping tablet replica
I20260806 02:50:41.053061 21370 raft_consensus.cc:2229] T cae06795f2824ec999a58e7357dbaf92 P 03a18a5ada5645d3840f0f8ea013b5da [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:50:41.053812 21370 raft_consensus.cc:2258] T cae06795f2824ec999a58e7357dbaf92 P 03a18a5ada5645d3840f0f8ea013b5da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:41.054776 21369 tablet_replica.cc:330] T cae06795f2824ec999a58e7357dbaf92 P 0dc2102de5404edf9cf061b729164162: stopping tablet replica
I20260806 02:50:41.055269 21369 raft_consensus.cc:2229] T cae06795f2824ec999a58e7357dbaf92 P 0dc2102de5404edf9cf061b729164162 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:50:41.056210 21369 raft_consensus.cc:2258] T cae06795f2824ec999a58e7357dbaf92 P 0dc2102de5404edf9cf061b729164162 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:41.056810 21370 ts_tablet_manager.cc:1905] T cae06795f2824ec999a58e7357dbaf92 P 03a18a5ada5645d3840f0f8ea013b5da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:41.058880 21368 ts_tablet_manager.cc:1905] T cae06795f2824ec999a58e7357dbaf92 P 476b7d1d0f4f46f2b896125fccfd3ec1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:41.059669 21343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:41.046275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33236 (local address 127.1.210.65:40697)
0806 02:50:41.046481 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:41.046487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:41.046508 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:50:41.046574 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:41.046583 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:41.046732 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:41.046843 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:41.047917 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:41.049537 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:41.052481 (+ 2944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:41.053198 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:41.053310 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:41.056864 (+ 3554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:41.056876 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:41.056895 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:41.057730 (+ 835us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:41.057800 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:41.057831 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:41.057929 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:41.059151 (+ 1222us) server_negotiation.cc:299] Negotiation successful
0806 02:50:41.059304 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 02:50:41.061236 21058 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:50:41.066282 21370 ts_tablet_manager.cc:1918] T cae06795f2824ec999a58e7357dbaf92 P 03a18a5ada5645d3840f0f8ea013b5da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:50:41.066530 21370 log.cc:1196] T cae06795f2824ec999a58e7357dbaf92 P 03a18a5ada5645d3840f0f8ea013b5da: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-1/wal/wals/cae06795f2824ec999a58e7357dbaf92
I20260806 02:50:41.067651 21370 ts_tablet_manager.cc:1939] T cae06795f2824ec999a58e7357dbaf92 P 03a18a5ada5645d3840f0f8ea013b5da: Deleting consensus metadata
I20260806 02:50:41.069258 21368 ts_tablet_manager.cc:1918] T cae06795f2824ec999a58e7357dbaf92 P 476b7d1d0f4f46f2b896125fccfd3ec1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:50:41.069522 21368 log.cc:1196] T cae06795f2824ec999a58e7357dbaf92 P 476b7d1d0f4f46f2b896125fccfd3ec1: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984304136196-1865-0/minicluster-data/ts-2/wal/wals/cae06795f2824ec999a58e7357dbaf92
I20260806 02:50:41.069644 20856 catalog_manager.cc:4841] TS 03a18a5ada5645d3840f0f8ea013b5da (127.1.210.66:46611): tablet cae06795f2824ec999a58e7357dbaf92 (table test-table [id=a36a4057de9f42f8b48fd3f12a84ca2c]) successfully deleted
I20260806 02:50:41.070377 21368 ts_tablet_manager.cc:1939] T cae06795f2824ec999a58e7357dbaf92 P 476b7d1d0f4f46f2b896125fccfd3ec1: Deleting consensus metadata
I20260806 02:50:41.072423 20857 catalog_manager.cc:4841] TS 476b7d1d0f4f46f2b896125fccfd3ec1 (127.1.210.67:44301): tablet cae06795f2824ec999a58e7357dbaf92 (table test-table [id=a36a4057de9f42f8b48fd3f12a84ca2c]) successfully deleted
I20260806 02:50:41.205137 21369 ts_tablet_manager.cc:1905] T cae06795f2824ec999a58e7357dbaf92 P 0dc2102de5404edf9cf061b729164162: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:41.207839 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 20933
W20260806 02:50:41.236457 20856 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.210.65:40697 (error 108)
I20260806 02:50:41.236676 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 21067
I20260806 02:50:41.239053 21375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:41.238203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:40697 (local address 127.0.0.1:33238)
0806 02:50:41.238668 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:41.238861 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:40697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":249,"threads_started":1}
W20260806 02:50:41.239557 20856 catalog_manager.cc:4584] TS 0dc2102de5404edf9cf061b729164162 (127.1.210.65:40697): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cae06795f2824ec999a58e7357dbaf92: Network error: Client connection negotiation failed: client connection to 127.1.210.65:40697: connect: Connection refused (error 111)
I20260806 02:50:41.245774 21340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:41.237952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51932 (local address 127.1.210.66:46611)
0806 02:50:41.238153 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:41.238159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:41.238179 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:50:41.238247 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:41.238252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:41.238373 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:41.238476 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:41.239392 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:41.240585 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:41.241965 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:41.242657 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:41.242836 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:41.243985 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:41.243999 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:41.244018 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:41.244900 (+ 882us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:41.244981 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:41.245006 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:41.245115 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:41.245283 (+ 168us) server_negotiation.cc:299] Negotiation successful
0806 02:50:41.245397 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 02:50:41.246587 21190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:50:41.293268 21375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:41.292710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:40697 (local address 127.0.0.1:33240)
0806 02:50:41.292959 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:41.293076 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:40697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260806 02:50:41.294487 21375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:41.294010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:40697 (local address 127.0.0.1:33254)
0806 02:50:41.294205 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:41.294293 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:40697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260806 02:50:41.355453 21375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:41.354861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:40697 (local address 127.0.0.1:33260)
0806 02:50:41.355109 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:41.355237 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:40697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260806 02:50:41.356693 21375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:41.356221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:40697 (local address 127.0.0.1:33274)
0806 02:50:41.356416 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:41.356508 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:40697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260806 02:50:41.364085 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 21067
I20260806 02:50:41.388121 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 21199
I20260806 02:50:41.399892 21342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:41.388973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43280 (local address 127.1.210.67:44301)
0806 02:50:41.389159 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:41.389171 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:41.389209 (+ 38us) server_negotiation.cc:407] Connection header received
0806 02:50:41.389379 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:41.389383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:41.389478 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:41.389568 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:41.389980 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:41.390791 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:41.392070 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:41.392481 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:41.392579 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:41.398166 (+ 5587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:41.398178 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:41.398194 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:41.399051 (+ 857us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:41.399117 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:41.399149 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:41.399253 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:41.399391 (+ 138us) server_negotiation.cc:299] Negotiation successful
0806 02:50:41.399523 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 02:50:41.400578 21322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:50:41.437227 21375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:41.436601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:40697 (local address 127.0.0.1:33290)
0806 02:50:41.436876 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:41.437000 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:40697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260806 02:50:41.438603 21375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:41.438049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:40697 (local address 127.0.0.1:33298)
0806 02:50:41.438316 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:41.438411 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:40697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260806 02:50:41.493530 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 21199
I20260806 02:50:41.511885 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 20840
I20260806 02:50:41.513242 20902 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:50:41.627960 21375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:41.627270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:40697 (local address 127.0.0.1:33302)
0806 02:50:41.627655 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:41.627762 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:40697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238}
I20260806 02:50:41.627981 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 20840
2026-08-06T02:50:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3254 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:50:41 dist-test-slave-8tdl krb5kdc[21381](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:50:41 dist-test-slave-8tdl krb5kdc[21381](info): set up 2 sockets
Aug 06 02:50:41 dist-test-slave-8tdl krb5kdc[21381](info): commencing operation
krb5kdc: starting...
W20260806 02:50:42.822309 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.004s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:50:43 dist-test-slave-8tdl krb5kdc[21381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:50:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
I20260806 02:50:43.159186 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:40905
--webserver_interface=127.1.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:33275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:40905
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:43.547967 21394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:43.548314 21394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:43.548415 21394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:43.556052 21394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:43.556195 21394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:43.556246 21394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:43.556298 21394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:43.556346 21394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:43.567763 21394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:33275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:40905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:40905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:50:43.569450 21394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:43.570914 21394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:43.585155 21400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:43.585120 21403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:43.585091 21401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:43.585882 21394 server_base.cc:1034] running on GCE node
I20260806 02:50:43.587234 21394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:43.589246 21394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:43.590559 21394 hybrid_clock.cc:648] HybridClock initialized: now 1785984643590482 us; error 68 us; skew 500 ppm
I20260806 02:50:43.591064 21394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:50:43 dist-test-slave-8tdl krb5kdc[21381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984643, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:43.595602 21394 init.cc:377] Logged in from keytab as kudu/127.1.210.126@KRBTEST.COM (short username kudu)
I20260806 02:50:43.597460 21394 webserver.cc:460] Webserver started at http://127.1.210.126:45573/ using document root <none> and password file <none>
I20260806 02:50:43.598274 21394 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:43.598402 21394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:43.598819 21394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:43.602321 21394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "7adf156c12dc48febf4bd1ff70cd429b"
format_stamp: "Formatted at 2026-08-06 02:50:43 on dist-test-slave-8tdl"
server_key: "ada1f16ea202244320b310b6b8df21ec"
server_key_iv: "a2bdfd7c4cfac4571a041a649719d19c"
server_key_version: "encryptionkey@0"
I20260806 02:50:43.603576 21394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "7adf156c12dc48febf4bd1ff70cd429b"
format_stamp: "Formatted at 2026-08-06 02:50:43 on dist-test-slave-8tdl"
server_key: "ada1f16ea202244320b310b6b8df21ec"
server_key_iv: "a2bdfd7c4cfac4571a041a649719d19c"
server_key_version: "encryptionkey@0"
I20260806 02:50:43.610564 21394 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260806 02:50:43.615676 21410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:43.617514 21394 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260806 02:50:43.617784 21394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "7adf156c12dc48febf4bd1ff70cd429b"
format_stamp: "Formatted at 2026-08-06 02:50:43 on dist-test-slave-8tdl"
server_key: "ada1f16ea202244320b310b6b8df21ec"
server_key_iv: "a2bdfd7c4cfac4571a041a649719d19c"
server_key_version: "encryptionkey@0"
I20260806 02:50:43.618058 21394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:43.650331 21394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:43.655260 21394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:43.655601 21394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:43.677305 21394 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:40905
I20260806 02:50:43.677335 21462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:40905 every 8 connection(s)
I20260806 02:50:43.678915 21394 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:43.683490 21463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:43.684019 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 21394
I20260806 02:50:43.684252 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:50:43.684700 1865 external_mini_cluster.cc:1428] Setting key 878bdb4488280e690a993a9c92f50bc6
I20260806 02:50:43.700908 21463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7adf156c12dc48febf4bd1ff70cd429b: Bootstrap starting.
Aug 06 02:50:43 dist-test-slave-8tdl krb5kdc[21381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:50:43.705296 21463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7adf156c12dc48febf4bd1ff70cd429b: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:43.707124 21463 log.cc:824] T 00000000000000000000000000000000 P 7adf156c12dc48febf4bd1ff70cd429b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:43.711680 21463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7adf156c12dc48febf4bd1ff70cd429b: No bootstrap required, opened a new log
I20260806 02:50:43.714284 21466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:43.686938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37496 (local address 127.1.210.126:40905)
0806 02:50:43.687936 (+ 998us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:43.687956 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:43.688014 (+ 58us) server_negotiation.cc:407] Connection header received
0806 02:50:43.689983 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:43.690026 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:43.690700 (+ 674us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:43.691292 (+ 592us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:43.692822 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:43.695052 (+ 2230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:43.696710 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:43.698056 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:43.698288 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:43.703243 (+ 4955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:43.703256 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:43.703280 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:43.703336 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:43.708399 (+ 5063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:43.709491 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:43.709501 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:43.709511 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:43.709772 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:43.710414 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:43.710422 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:43.710426 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:43.711326 (+ 900us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:43.711693 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:43.712431 (+ 738us) server_negotiation.cc:299] Negotiation successful
0806 02:50:43.713083 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":470,"threads_started":1}
I20260806 02:50:43.722141 21463 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7adf156c12dc48febf4bd1ff70cd429b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7adf156c12dc48febf4bd1ff70cd429b" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 40905 } }
I20260806 02:50:43.722853 21463 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7adf156c12dc48febf4bd1ff70cd429b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:43.722987 21463 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7adf156c12dc48febf4bd1ff70cd429b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7adf156c12dc48febf4bd1ff70cd429b, State: Initialized, Role: FOLLOWER
I20260806 02:50:43.723915 21463 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7adf156c12dc48febf4bd1ff70cd429b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7adf156c12dc48febf4bd1ff70cd429b" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 40905 } }
I20260806 02:50:43.724352 21463 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7adf156c12dc48febf4bd1ff70cd429b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:50:43.724530 21463 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7adf156c12dc48febf4bd1ff70cd429b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:50:43.724761 21463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7adf156c12dc48febf4bd1ff70cd429b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:43.728912 21463 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7adf156c12dc48febf4bd1ff70cd429b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7adf156c12dc48febf4bd1ff70cd429b" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 40905 } }
I20260806 02:50:43.729640 21463 leader_election.cc:302] T 00000000000000000000000000000000 P 7adf156c12dc48febf4bd1ff70cd429b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7adf156c12dc48febf4bd1ff70cd429b; no voters:
I20260806 02:50:43.730481 21463 leader_election.cc:288] T 00000000000000000000000000000000 P 7adf156c12dc48febf4bd1ff70cd429b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:50:43.730635 21468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7adf156c12dc48febf4bd1ff70cd429b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:43.731842 21468 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7adf156c12dc48febf4bd1ff70cd429b [term 1 LEADER]: Becoming Leader. State: Replica: 7adf156c12dc48febf4bd1ff70cd429b, State: Running, Role: LEADER
I20260806 02:50:43.732841 21468 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7adf156c12dc48febf4bd1ff70cd429b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7adf156c12dc48febf4bd1ff70cd429b" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 40905 } }
I20260806 02:50:43.733750 21463 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7adf156c12dc48febf4bd1ff70cd429b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:43.736953 21469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7adf156c12dc48febf4bd1ff70cd429b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7adf156c12dc48febf4bd1ff70cd429b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7adf156c12dc48febf4bd1ff70cd429b" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 40905 } } }
I20260806 02:50:43.737052 21471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7adf156c12dc48febf4bd1ff70cd429b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7adf156c12dc48febf4bd1ff70cd429b. Latest consensus state: current_term: 1 leader_uuid: "7adf156c12dc48febf4bd1ff70cd429b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7adf156c12dc48febf4bd1ff70cd429b" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 40905 } } }
I20260806 02:50:43.737445 21469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7adf156c12dc48febf4bd1ff70cd429b [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:43.737501 21471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7adf156c12dc48febf4bd1ff70cd429b [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:43.741008 21474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:50:43.751087 21474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:43.762398 21474 catalog_manager.cc:1347] Generated new cluster ID: 33092b24f1c249f884650328e531847c
I20260806 02:50:43.762544 21474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:43.777908 21474 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:50:43.779960 21474 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:43.799356 21474 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7adf156c12dc48febf4bd1ff70cd429b: Generated new TSK 0
I20260806 02:50:43.800487 21474 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.210.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab.
Entry for principal kudu/127.1.210.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab.
I20260806 02:50:43.887379 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.65:0
--local_ip_for_outbound_sockets=127.1.210.65
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:40905
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:33275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:44.281179 21490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:44.281509 21490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:44.281615 21490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:44.289206 21490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:44.289335 21490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:44.289455 21490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.65
I20260806 02:50:44.300341 21490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:33275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.65.keytab
--principal=kudu/127.1.210.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:40905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.65
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:50:44.301775 21490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:44.303390 21490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:44.318361 21495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:44.318384 21498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:44.318617 21496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:44.320091 21490 server_base.cc:1034] running on GCE node
I20260806 02:50:44.320879 21490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:44.322109 21490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:44.323405 21490 hybrid_clock.cc:648] HybridClock initialized: now 1785984644323338 us; error 69 us; skew 500 ppm
I20260806 02:50:44.323817 21490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:50:44 dist-test-slave-8tdl krb5kdc[21381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984644, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:44.327939 21490 init.cc:377] Logged in from keytab as kudu/127.1.210.65@KRBTEST.COM (short username kudu)
I20260806 02:50:44.329810 21490 webserver.cc:460] Webserver started at http://127.1.210.65:38225/ using document root <none> and password file <none>
I20260806 02:50:44.330628 21490 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:44.330856 21490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:44.331229 21490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:44.334511 21490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-0/data/instance:
uuid: "329a80a10d7f4ee9ae05c2e18048c77d"
format_stamp: "Formatted at 2026-08-06 02:50:44 on dist-test-slave-8tdl"
server_key: "3273dd8b30f7b2506a555fa728a448d5"
server_key_iv: "f254cdf71d4ad98e174fce6117277781"
server_key_version: "encryptionkey@0"
I20260806 02:50:44.335686 21490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance:
uuid: "329a80a10d7f4ee9ae05c2e18048c77d"
format_stamp: "Formatted at 2026-08-06 02:50:44 on dist-test-slave-8tdl"
server_key: "3273dd8b30f7b2506a555fa728a448d5"
server_key_iv: "f254cdf71d4ad98e174fce6117277781"
server_key_version: "encryptionkey@0"
I20260806 02:50:44.341974 21490 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260806 02:50:44.346364 21505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:44.348146 21490 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:50:44.348348 21490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-0/wal
uuid: "329a80a10d7f4ee9ae05c2e18048c77d"
format_stamp: "Formatted at 2026-08-06 02:50:44 on dist-test-slave-8tdl"
server_key: "3273dd8b30f7b2506a555fa728a448d5"
server_key_iv: "f254cdf71d4ad98e174fce6117277781"
server_key_version: "encryptionkey@0"
I20260806 02:50:44.348589 21490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:44.367728 21490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:44.372714 21490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:44.373061 21490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:44.374161 21490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:44.376214 21490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:44.376318 21490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:44.376467 21490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:44.376545 21490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:44.414007 21490 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.65:40561
I20260806 02:50:44.414058 21618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.65:40561 every 8 connection(s)
I20260806 02:50:44.415498 21490 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
I20260806 02:50:44.416941 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 21490
I20260806 02:50:44.417155 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance
I20260806 02:50:44.417636 1865 external_mini_cluster.cc:1428] Setting key 1859f7a11add987a407f758d028e62ff
Aug 06 02:50:44 dist-test-slave-8tdl krb5kdc[21381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984644, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.65@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:50:44.443567 21621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:44.418389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:35873 (local address 127.1.210.126:40905)
0806 02:50:44.419118 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:44.419125 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:44.420429 (+ 1304us) server_negotiation.cc:407] Connection header received
0806 02:50:44.423636 (+ 3207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:44.423640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:44.423746 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:44.423851 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:44.426311 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:44.427156 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:44.429908 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:44.430498 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:44.430634 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:44.435659 (+ 5025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:44.435669 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:44.435677 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:44.435733 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:44.439679 (+ 3946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:44.440746 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:44.440766 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:44.440772 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:44.440967 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:44.441483 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:44.441487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:44.441490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:44.441847 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:44.442018 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:44.442897 (+ 879us) server_negotiation.cc:299] Negotiation successful
0806 02:50:44.443166 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":294,"threads_started":1}
I20260806 02:50:44.444669 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:44.419175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:40905 (local address 127.1.210.65:35873)
0806 02:50:44.420224 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:44.420279 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:44.423280 (+ 3001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:44.424196 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:44.424212 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:44.425097 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:44.426073 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:44.426104 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:44.427346 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:44.427355 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:44.429687 (+ 2332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:44.429709 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:44.430825 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:44.430836 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:44.431218 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:44.432071 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:44.432127 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:44.435350 (+ 3223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:44.439897 (+ 4547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:44.439906 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:44.439922 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:44.440587 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:44.441092 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:44.441095 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:44.441098 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:44.441323 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:44.441992 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:44.442001 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:44.442414 (+ 413us) client_negotiation.cc:769] Sending connection context
0806 02:50:44.442803 (+ 389us) client_negotiation.cc:240] Negotiation successful
0806 02:50:44.443422 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":427,"threads_started":1}
I20260806 02:50:44.446552 21619 heartbeater.cc:347] Connected to a master server at 127.1.210.126:40905
I20260806 02:50:44.446936 21619 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:44.447803 21619 heartbeater.cc:511] Master 127.1.210.126:40905 requested a full tablet report, sending...
I20260806 02:50:44.450654 21427 ts_manager.cc:194] Registered new tserver with Master: 329a80a10d7f4ee9ae05c2e18048c77d (127.1.210.65:40561)
I20260806 02:50:44.453342 21427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.65@KRBTEST.COM'} at 127.1.210.65:35873
WARNING: no policy specified for kudu/127.1.210.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab.
Entry for principal kudu/127.1.210.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab.
I20260806 02:50:44.499349 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.66:0
--local_ip_for_outbound_sockets=127.1.210.66
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:40905
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:33275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:44.858840 21625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:44.859153 21625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:44.859308 21625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:44.866878 21625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:44.867024 21625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:44.867228 21625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.66
I20260806 02:50:44.877310 21625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:33275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.66.keytab
--principal=kudu/127.1.210.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:40905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.66
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:50:44.878734 21625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:44.880157 21625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:44.894532 21633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:44.894580 21631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:44.894637 21630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:44.900867 21625 server_base.cc:1034] running on GCE node
I20260806 02:50:44.901728 21625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:44.902908 21625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:44.904181 21625 hybrid_clock.cc:648] HybridClock initialized: now 1785984644904120 us; error 48 us; skew 500 ppm
I20260806 02:50:44.904568 21625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:50:44 dist-test-slave-8tdl krb5kdc[21381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984644, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:44.908141 21625 init.cc:377] Logged in from keytab as kudu/127.1.210.66@KRBTEST.COM (short username kudu)
I20260806 02:50:44.909936 21625 webserver.cc:460] Webserver started at http://127.1.210.66:42619/ using document root <none> and password file <none>
I20260806 02:50:44.910873 21625 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:44.911158 21625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:44.911581 21625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:44.915273 21625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-1/data/instance:
uuid: "2fa98bf3b2584963852d66c5bd93fea1"
format_stamp: "Formatted at 2026-08-06 02:50:44 on dist-test-slave-8tdl"
server_key: "c3e714dd30b047f422d81a4e54f1bf19"
server_key_iv: "75ef6ba06ad7e26ca275a27656a6d09d"
server_key_version: "encryptionkey@0"
I20260806 02:50:44.916240 21625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance:
uuid: "2fa98bf3b2584963852d66c5bd93fea1"
format_stamp: "Formatted at 2026-08-06 02:50:44 on dist-test-slave-8tdl"
server_key: "c3e714dd30b047f422d81a4e54f1bf19"
server_key_iv: "75ef6ba06ad7e26ca275a27656a6d09d"
server_key_version: "encryptionkey@0"
I20260806 02:50:44.922341 21625 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 02:50:44.926438 21640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:44.928064 21625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:50:44.928288 21625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-1/wal
uuid: "2fa98bf3b2584963852d66c5bd93fea1"
format_stamp: "Formatted at 2026-08-06 02:50:44 on dist-test-slave-8tdl"
server_key: "c3e714dd30b047f422d81a4e54f1bf19"
server_key_iv: "75ef6ba06ad7e26ca275a27656a6d09d"
server_key_version: "encryptionkey@0"
I20260806 02:50:44.928550 21625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:44.989656 21625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:44.994468 21625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:44.994851 21625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:44.995915 21625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:44.997371 21625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:44.997483 21625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:44.997637 21625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:44.997713 21625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:45.034816 21625 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.66:46771
I20260806 02:50:45.034946 21756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.66:46771 every 8 connection(s)
I20260806 02:50:45.036334 21625 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
I20260806 02:50:45.044428 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 21625
I20260806 02:50:45.044641 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance
I20260806 02:50:45.045328 1865 external_mini_cluster.cc:1428] Setting key e9cd3ef71a9a6dde08f230647edb9533
Aug 06 02:50:45 dist-test-slave-8tdl krb5kdc[21381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984644, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.66@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:50:45.061904 21759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:45.039362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:58933 (local address 127.1.210.126:40905)
0806 02:50:45.040002 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:45.040019 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:45.040859 (+ 840us) server_negotiation.cc:407] Connection header received
0806 02:50:45.043097 (+ 2238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:45.043103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:45.043212 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:45.043338 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:45.045756 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.046801 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.049128 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.049642 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.049769 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:45.054934 (+ 5165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:45.054940 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:45.054943 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:45.055026 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:45.057995 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:45.059000 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:45.059005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:45.059007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:45.059156 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:45.059672 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:45.059676 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:45.059678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:45.060122 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:45.060358 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:45.061225 (+ 867us) server_negotiation.cc:299] Negotiation successful
0806 02:50:45.061503 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":285,"threads_started":1}
I20260806 02:50:45.063166 21760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:45.039875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:40905 (local address 127.1.210.66:58933)
0806 02:50:45.040675 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:45.040724 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:45.042770 (+ 2046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:45.043635 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:45.043645 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:45.044524 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:45.045522 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:45.045555 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.046980 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.046984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:45.048936 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:45.048957 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.049921 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.049931 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:45.050291 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:45.051189 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:45.051227 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:45.054557 (+ 3330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:45.058196 (+ 3639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:45.058207 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:45.058230 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:45.058841 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:45.059286 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:45.059293 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:45.059296 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:45.059527 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:45.060311 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:45.060322 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:45.060793 (+ 471us) client_negotiation.cc:769] Sending connection context
0806 02:50:45.061114 (+ 321us) client_negotiation.cc:240] Negotiation successful
0806 02:50:45.061801 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":322,"threads_started":1}
I20260806 02:50:45.065145 21757 heartbeater.cc:347] Connected to a master server at 127.1.210.126:40905
I20260806 02:50:45.065562 21757 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:45.066390 21757 heartbeater.cc:511] Master 127.1.210.126:40905 requested a full tablet report, sending...
I20260806 02:50:45.068164 21427 ts_manager.cc:194] Registered new tserver with Master: 2fa98bf3b2584963852d66c5bd93fea1 (127.1.210.66:46771)
I20260806 02:50:45.069084 21427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.66@KRBTEST.COM'} at 127.1.210.66:58933
WARNING: no policy specified for kudu/127.1.210.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab.
Entry for principal kudu/127.1.210.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab.
I20260806 02:50:45.127491 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.67:0
--local_ip_for_outbound_sockets=127.1.210.67
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:40905
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:33275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:50:45.457260 21619 heartbeater.cc:503] Master 127.1.210.126:40905 was elected leader, sending a full tablet report...
W20260806 02:50:45.493641 21763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:45.494011 21763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:45.494202 21763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:45.501739 21763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:45.501919 21763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:45.502099 21763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.67
I20260806 02:50:45.512424 21763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:33275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.67.keytab
--principal=kudu/127.1.210.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:40905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.67
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:50:45.513852 21763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:45.515365 21763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:45.529475 21768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:45.529558 21771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:45.529510 21769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:45.530582 21763 server_base.cc:1034] running on GCE node
I20260806 02:50:45.531332 21763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:45.532328 21763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:45.533541 21763 hybrid_clock.cc:648] HybridClock initialized: now 1785984645533478 us; error 54 us; skew 500 ppm
I20260806 02:50:45.534011 21763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:50:45 dist-test-slave-8tdl krb5kdc[21381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984645, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:45.537567 21763 init.cc:377] Logged in from keytab as kudu/127.1.210.67@KRBTEST.COM (short username kudu)
I20260806 02:50:45.539376 21763 webserver.cc:460] Webserver started at http://127.1.210.67:38493/ using document root <none> and password file <none>
I20260806 02:50:45.540099 21763 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:45.540205 21763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:45.540568 21763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:45.544026 21763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-2/data/instance:
uuid: "69b4df6ad612461b8fca75d7f0293a4b"
format_stamp: "Formatted at 2026-08-06 02:50:45 on dist-test-slave-8tdl"
server_key: "07722ef9f3c3c5fede59af2b20ffd2f4"
server_key_iv: "7addaef80080ca4471752b0b832164da"
server_key_version: "encryptionkey@0"
I20260806 02:50:45.544888 21763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance:
uuid: "69b4df6ad612461b8fca75d7f0293a4b"
format_stamp: "Formatted at 2026-08-06 02:50:45 on dist-test-slave-8tdl"
server_key: "07722ef9f3c3c5fede59af2b20ffd2f4"
server_key_iv: "7addaef80080ca4471752b0b832164da"
server_key_version: "encryptionkey@0"
I20260806 02:50:45.550462 21763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260806 02:50:45.554668 21778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:45.556550 21763 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:50:45.556777 21763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-2/wal
uuid: "69b4df6ad612461b8fca75d7f0293a4b"
format_stamp: "Formatted at 2026-08-06 02:50:45 on dist-test-slave-8tdl"
server_key: "07722ef9f3c3c5fede59af2b20ffd2f4"
server_key_iv: "7addaef80080ca4471752b0b832164da"
server_key_version: "encryptionkey@0"
I20260806 02:50:45.557015 21763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:45.573490 21763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:45.578717 21763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:45.578990 21763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:45.580034 21763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:45.581396 21763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:45.581471 21763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:45.581609 21763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:45.581684 21763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:45.616456 21763 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.67:41485
I20260806 02:50:45.616557 21892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.67:41485 every 8 connection(s)
I20260806 02:50:45.617830 21763 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
I20260806 02:50:45.622879 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 21763
I20260806 02:50:45.623071 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance
I20260806 02:50:45.623623 1865 external_mini_cluster.cc:1428] Setting key 2d5804d3d9e9efd4f47385010ad5f8de
Aug 06 02:50:45 dist-test-slave-8tdl krb5kdc[21381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984645, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.67@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:50:45.641651 21895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:45.620757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:55417 (local address 127.1.210.126:40905)
0806 02:50:45.621235 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:45.621242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:45.622521 (+ 1279us) server_negotiation.cc:407] Connection header received
0806 02:50:45.625321 (+ 2800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:45.625325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:45.625423 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:45.625549 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:45.627687 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.628472 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.630205 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.630602 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.630799 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:45.634966 (+ 4167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:45.634971 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:45.634974 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:45.635019 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:45.638052 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:45.639105 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:45.639110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:45.639113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:45.639291 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:45.639836 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:45.639840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:45.639842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:45.640167 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:45.640300 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:45.641146 (+ 846us) server_negotiation.cc:299] Negotiation successful
0806 02:50:45.641343 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":215,"threads_started":1}
I20260806 02:50:45.642649 21896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:45.621415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:40905 (local address 127.1.210.67:55417)
0806 02:50:45.622322 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:45.622374 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:45.624742 (+ 2368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:45.625859 (+ 1117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:45.625873 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:45.626806 (+ 933us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:45.627523 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:45.627544 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.628619 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.628623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:45.630065 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:45.630079 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.630885 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.630890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:45.631166 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:45.631935 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:45.631961 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:45.634725 (+ 2764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:45.638277 (+ 3552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:45.638287 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:45.638309 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:45.638975 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:45.639447 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:45.639452 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:45.639455 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:45.639690 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:45.640300 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:45.640307 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:45.640775 (+ 468us) client_negotiation.cc:769] Sending connection context
0806 02:50:45.641025 (+ 250us) client_negotiation.cc:240] Negotiation successful
0806 02:50:45.641462 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":337,"threads_started":1}
I20260806 02:50:45.644280 21893 heartbeater.cc:347] Connected to a master server at 127.1.210.126:40905
I20260806 02:50:45.644649 21893 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:45.645432 21893 heartbeater.cc:511] Master 127.1.210.126:40905 requested a full tablet report, sending...
I20260806 02:50:45.647194 21427 ts_manager.cc:194] Registered new tserver with Master: 69b4df6ad612461b8fca75d7f0293a4b (127.1.210.67:41485)
I20260806 02:50:45.648066 21427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.210.67@KRBTEST.COM'} at 127.1.210.67:55417
I20260806 02:50:45.652200 1865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:50:45.671345 21895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:45.656775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53826 (local address 127.1.210.126:40905)
0806 02:50:45.656959 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:45.656964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:45.657525 (+ 561us) server_negotiation.cc:407] Connection header received
0806 02:50:45.657743 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:45.657750 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:45.657830 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:45.657941 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:45.659530 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.660283 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.661733 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.662259 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.662400 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:45.665219 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:45.665225 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:45.665229 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:45.665268 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:45.668409 (+ 3141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:45.669156 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:45.669162 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:45.669166 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:45.669305 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:45.669715 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:45.669722 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:45.669730 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:45.670084 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:45.670344 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:45.670758 (+ 414us) server_negotiation.cc:299] Negotiation successful
0806 02:50:45.671019 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:50:45.676203 21427 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:50:45.683930 21427 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:50:45.723253 21907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:45.697687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:46771 (local address 127.0.0.1:38814)
0806 02:50:45.711725 (+ 14038us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:45.711749 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:45.711898 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:45.712715 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:45.712723 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:45.712753 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:45.713557 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:45.713580 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.715938 (+ 2358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.715945 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:45.719902 (+ 3957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:45.719923 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.722319 (+ 2396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.722324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:45.722498 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:45.722527 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 02:50:45.722603 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 02:50:45.722738 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13894,"mutex_wait_us":336,"thread_start_us":12631,"threads_started":1}
I20260806 02:50:45.723278 21906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:45.697563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:41485 (local address 127.0.0.1:40184)
0806 02:50:45.709700 (+ 12137us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:45.709770 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:45.710041 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:45.712143 (+ 2102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:45.712155 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:45.712200 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:45.713662 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:45.713674 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.715856 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.715888 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:45.719902 (+ 4014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:45.719923 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.722027 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.722034 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:45.722244 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:45.722328 (+ 84us) client_negotiation.cc:769] Sending connection context
0806 02:50:45.722465 (+ 137us) client_negotiation.cc:240] Negotiation successful
0806 02:50:45.722705 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11740,"mutex_wait_us":432,"thread_start_us":13834,"threads_started":1}
I20260806 02:50:45.724145 21909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:45.698314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38814 (local address 127.1.210.66:46771)
0806 02:50:45.712010 (+ 13696us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:45.712021 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:45.712055 (+ 34us) server_negotiation.cc:407] Connection header received
0806 02:50:45.712166 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:45.712175 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:45.712463 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:45.713254 (+ 791us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:45.714358 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.715795 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.720161 (+ 4366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.721766 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.721922 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:45.722170 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:45.723564 (+ 1394us) server_negotiation.cc:299] Negotiation successful
0806 02:50:45.723785 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13510,"thread_start_us":13389,"threads_started":1}
I20260806 02:50:45.724413 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:45.697801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40184 (local address 127.1.210.67:41485)
0806 02:50:45.710910 (+ 13109us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:45.710923 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:45.710965 (+ 42us) server_negotiation.cc:407] Connection header received
0806 02:50:45.711080 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:45.711091 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:45.711392 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:45.712823 (+ 1431us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:45.714002 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.715660 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.720161 (+ 4501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.721765 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.723537 (+ 1772us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:45.723797 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:45.723867 (+ 70us) server_negotiation.cc:299] Negotiation successful
0806 02:50:45.724047 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12853,"thread_start_us":13757,"threads_started":1}
I20260806 02:50:45.726092 21910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:45.700035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:40561 (local address 127.0.0.1:57006)
0806 02:50:45.714234 (+ 14199us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:45.714255 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:45.714372 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:45.715536 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:45.715541 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:45.715567 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:45.716104 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:45.716117 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.717792 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.717799 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:45.720434 (+ 2635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:45.720456 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.725223 (+ 4767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.725230 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:45.725409 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:45.725460 (+ 51us) client_negotiation.cc:769] Sending connection context
0806 02:50:45.725563 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 02:50:45.725708 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14083,"mutex_wait_us":1,"thread_start_us":13647,"threads_started":1}
I20260806 02:50:45.727077 21911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:45.700722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57006 (local address 127.1.210.65:40561)
0806 02:50:45.712520 (+ 11798us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:45.712528 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:45.714758 (+ 2230us) server_negotiation.cc:407] Connection header received
0806 02:50:45.715061 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:45.715069 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:45.715322 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:45.715520 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:45.716309 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.717609 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.720676 (+ 3067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.724954 (+ 4278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.725165 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:45.725461 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:45.726521 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0806 02:50:45.726750 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11612,"thread_start_us":11472,"threads_started":1}
I20260806 02:50:45.728938 21691 tablet_service.cc:1467] Processing CreateTablet for tablet 9a3d69b27f22403bbc708ff5842b2878 (DEFAULT_TABLE table=test-table [id=cc875474ee70416e92262036d75fb387]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:45.730616 21691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a3d69b27f22403bbc708ff5842b2878. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:45.732164 21554 tablet_service.cc:1467] Processing CreateTablet for tablet 9a3d69b27f22403bbc708ff5842b2878 (DEFAULT_TABLE table=test-table [id=cc875474ee70416e92262036d75fb387]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:45.733546 21554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a3d69b27f22403bbc708ff5842b2878. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:45.737504 21827 tablet_service.cc:1467] Processing CreateTablet for tablet 9a3d69b27f22403bbc708ff5842b2878 (DEFAULT_TABLE table=test-table [id=cc875474ee70416e92262036d75fb387]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:45.738991 21827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a3d69b27f22403bbc708ff5842b2878. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:45.744421 21912 tablet_bootstrap.cc:492] T 9a3d69b27f22403bbc708ff5842b2878 P 2fa98bf3b2584963852d66c5bd93fea1: Bootstrap starting.
I20260806 02:50:45.747969 21912 tablet_bootstrap.cc:654] T 9a3d69b27f22403bbc708ff5842b2878 P 2fa98bf3b2584963852d66c5bd93fea1: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:45.749138 21912 log.cc:824] T 9a3d69b27f22403bbc708ff5842b2878 P 2fa98bf3b2584963852d66c5bd93fea1: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:45.752473 21912 tablet_bootstrap.cc:492] T 9a3d69b27f22403bbc708ff5842b2878 P 2fa98bf3b2584963852d66c5bd93fea1: No bootstrap required, opened a new log
I20260806 02:50:45.752784 21912 ts_tablet_manager.cc:1397] T 9a3d69b27f22403bbc708ff5842b2878 P 2fa98bf3b2584963852d66c5bd93fea1: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260806 02:50:45.755995 21915 tablet_bootstrap.cc:492] T 9a3d69b27f22403bbc708ff5842b2878 P 329a80a10d7f4ee9ae05c2e18048c77d: Bootstrap starting.
I20260806 02:50:45.757972 21914 tablet_bootstrap.cc:492] T 9a3d69b27f22403bbc708ff5842b2878 P 69b4df6ad612461b8fca75d7f0293a4b: Bootstrap starting.
I20260806 02:50:45.760149 21915 tablet_bootstrap.cc:654] T 9a3d69b27f22403bbc708ff5842b2878 P 329a80a10d7f4ee9ae05c2e18048c77d: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:45.761046 21914 tablet_bootstrap.cc:654] T 9a3d69b27f22403bbc708ff5842b2878 P 69b4df6ad612461b8fca75d7f0293a4b: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:45.761488 21915 log.cc:824] T 9a3d69b27f22403bbc708ff5842b2878 P 329a80a10d7f4ee9ae05c2e18048c77d: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:45.762264 21914 log.cc:824] T 9a3d69b27f22403bbc708ff5842b2878 P 69b4df6ad612461b8fca75d7f0293a4b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:45.763983 21912 raft_consensus.cc:348] T 9a3d69b27f22403bbc708ff5842b2878 P 2fa98bf3b2584963852d66c5bd93fea1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69b4df6ad612461b8fca75d7f0293a4b" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41485 } } peers { permanent_uuid: "2fa98bf3b2584963852d66c5bd93fea1" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 46771 } } peers { permanent_uuid: "329a80a10d7f4ee9ae05c2e18048c77d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 40561 } }
I20260806 02:50:45.764418 21912 raft_consensus.cc:374] T 9a3d69b27f22403bbc708ff5842b2878 P 2fa98bf3b2584963852d66c5bd93fea1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:45.764531 21912 raft_consensus.cc:729] T 9a3d69b27f22403bbc708ff5842b2878 P 2fa98bf3b2584963852d66c5bd93fea1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fa98bf3b2584963852d66c5bd93fea1, State: Initialized, Role: FOLLOWER
I20260806 02:50:45.765252 21912 consensus_queue.cc:260] T 9a3d69b27f22403bbc708ff5842b2878 P 2fa98bf3b2584963852d66c5bd93fea1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69b4df6ad612461b8fca75d7f0293a4b" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41485 } } peers { permanent_uuid: "2fa98bf3b2584963852d66c5bd93fea1" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 46771 } } peers { permanent_uuid: "329a80a10d7f4ee9ae05c2e18048c77d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 40561 } }
I20260806 02:50:45.766664 21757 heartbeater.cc:503] Master 127.1.210.126:40905 was elected leader, sending a full tablet report...
I20260806 02:50:45.767316 21915 tablet_bootstrap.cc:492] T 9a3d69b27f22403bbc708ff5842b2878 P 329a80a10d7f4ee9ae05c2e18048c77d: No bootstrap required, opened a new log
I20260806 02:50:45.767349 21912 ts_tablet_manager.cc:1428] T 9a3d69b27f22403bbc708ff5842b2878 P 2fa98bf3b2584963852d66c5bd93fea1: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
I20260806 02:50:45.767639 21915 ts_tablet_manager.cc:1397] T 9a3d69b27f22403bbc708ff5842b2878 P 329a80a10d7f4ee9ae05c2e18048c77d: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260806 02:50:45.771987 21914 tablet_bootstrap.cc:492] T 9a3d69b27f22403bbc708ff5842b2878 P 69b4df6ad612461b8fca75d7f0293a4b: No bootstrap required, opened a new log
I20260806 02:50:45.772295 21914 ts_tablet_manager.cc:1397] T 9a3d69b27f22403bbc708ff5842b2878 P 69b4df6ad612461b8fca75d7f0293a4b: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.005s
I20260806 02:50:45.777021 21915 raft_consensus.cc:348] T 9a3d69b27f22403bbc708ff5842b2878 P 329a80a10d7f4ee9ae05c2e18048c77d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69b4df6ad612461b8fca75d7f0293a4b" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41485 } } peers { permanent_uuid: "2fa98bf3b2584963852d66c5bd93fea1" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 46771 } } peers { permanent_uuid: "329a80a10d7f4ee9ae05c2e18048c77d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 40561 } }
I20260806 02:50:45.777462 21915 raft_consensus.cc:374] T 9a3d69b27f22403bbc708ff5842b2878 P 329a80a10d7f4ee9ae05c2e18048c77d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:45.777566 21915 raft_consensus.cc:729] T 9a3d69b27f22403bbc708ff5842b2878 P 329a80a10d7f4ee9ae05c2e18048c77d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 329a80a10d7f4ee9ae05c2e18048c77d, State: Initialized, Role: FOLLOWER
I20260806 02:50:45.778244 21915 consensus_queue.cc:260] T 9a3d69b27f22403bbc708ff5842b2878 P 329a80a10d7f4ee9ae05c2e18048c77d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69b4df6ad612461b8fca75d7f0293a4b" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41485 } } peers { permanent_uuid: "2fa98bf3b2584963852d66c5bd93fea1" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 46771 } } peers { permanent_uuid: "329a80a10d7f4ee9ae05c2e18048c77d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 40561 } }
I20260806 02:50:45.780169 21915 ts_tablet_manager.cc:1428] T 9a3d69b27f22403bbc708ff5842b2878 P 329a80a10d7f4ee9ae05c2e18048c77d: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260806 02:50:45.781134 21914 raft_consensus.cc:348] T 9a3d69b27f22403bbc708ff5842b2878 P 69b4df6ad612461b8fca75d7f0293a4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69b4df6ad612461b8fca75d7f0293a4b" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41485 } } peers { permanent_uuid: "2fa98bf3b2584963852d66c5bd93fea1" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 46771 } } peers { permanent_uuid: "329a80a10d7f4ee9ae05c2e18048c77d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 40561 } }
I20260806 02:50:45.781546 21914 raft_consensus.cc:374] T 9a3d69b27f22403bbc708ff5842b2878 P 69b4df6ad612461b8fca75d7f0293a4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:45.781656 21914 raft_consensus.cc:729] T 9a3d69b27f22403bbc708ff5842b2878 P 69b4df6ad612461b8fca75d7f0293a4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69b4df6ad612461b8fca75d7f0293a4b, State: Initialized, Role: FOLLOWER
I20260806 02:50:45.782330 21914 consensus_queue.cc:260] T 9a3d69b27f22403bbc708ff5842b2878 P 69b4df6ad612461b8fca75d7f0293a4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69b4df6ad612461b8fca75d7f0293a4b" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41485 } } peers { permanent_uuid: "2fa98bf3b2584963852d66c5bd93fea1" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 46771 } } peers { permanent_uuid: "329a80a10d7f4ee9ae05c2e18048c77d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 40561 } }
I20260806 02:50:45.783495 21893 heartbeater.cc:503] Master 127.1.210.126:40905 was elected leader, sending a full tablet report...
I20260806 02:50:45.784130 21914 ts_tablet_manager.cc:1428] T 9a3d69b27f22403bbc708ff5842b2878 P 69b4df6ad612461b8fca75d7f0293a4b: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
W20260806 02:50:45.788769 21758 tablet.cc:2369] T 9a3d69b27f22403bbc708ff5842b2878 P 2fa98bf3b2584963852d66c5bd93fea1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:50:45.801770 21920 raft_consensus.cc:482] T 9a3d69b27f22403bbc708ff5842b2878 P 69b4df6ad612461b8fca75d7f0293a4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:50:45.802161 21920 raft_consensus.cc:504] T 9a3d69b27f22403bbc708ff5842b2878 P 69b4df6ad612461b8fca75d7f0293a4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69b4df6ad612461b8fca75d7f0293a4b" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41485 } } peers { permanent_uuid: "2fa98bf3b2584963852d66c5bd93fea1" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 46771 } } peers { permanent_uuid: "329a80a10d7f4ee9ae05c2e18048c77d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 40561 } }
I20260806 02:50:45.804013 21920 leader_election.cc:288] T 9a3d69b27f22403bbc708ff5842b2878 P 69b4df6ad612461b8fca75d7f0293a4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fa98bf3b2584963852d66c5bd93fea1 (127.1.210.66:46771), 329a80a10d7f4ee9ae05c2e18048c77d (127.1.210.65:40561)
I20260806 02:50:45.811169 21909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:45.804498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:45611 (local address 127.1.210.66:46771)
0806 02:50:45.804701 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:45.804708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:45.804733 (+ 25us) server_negotiation.cc:407] Connection header received
0806 02:50:45.805059 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:45.805064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:45.805130 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:45.805267 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:45.805714 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.806791 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.808972 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.809999 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.810153 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:45.810230 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:45.810750 (+ 520us) server_negotiation.cc:299] Negotiation successful
0806 02:50:45.810861 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:50:45.811205 21896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:45.804435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:46771 (local address 127.1.210.67:45611)
0806 02:50:45.804634 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:45.804663 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:45.804796 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:45.805294 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:45.805298 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:45.805327 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:45.805562 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:45.805571 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.806919 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.806923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:45.808801 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:45.808817 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.810326 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.810333 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:45.810536 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:45.810573 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 02:50:45.810720 (+ 147us) client_negotiation.cc:240] Negotiation successful
0806 02:50:45.810865 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260806 02:50:45.811901 21911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:45.804497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:33215 (local address 127.1.210.65:40561)
0806 02:50:45.804702 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:45.804708 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:45.805833 (+ 1125us) server_negotiation.cc:407] Connection header received
0806 02:50:45.805919 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:45.805924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:45.806005 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:45.806106 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:45.806579 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.807408 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.809527 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.810861 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.811018 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:45.811100 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:45.811536 (+ 436us) server_negotiation.cc:299] Negotiation successful
0806 02:50:45.811683 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:50:45.811964 21921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:45.804458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:40561 (local address 127.1.210.67:33215)
0806 02:50:45.805722 (+ 1264us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:45.805736 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:45.805829 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:45.806123 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:45.806126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:45.806144 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:45.806417 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:45.806424 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.807563 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.807569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:45.809362 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:45.809373 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.811074 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.811080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:45.811286 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:45.811321 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 02:50:45.811470 (+ 149us) client_negotiation.cc:240] Negotiation successful
0806 02:50:45.811599 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1164,"mutex_wait_us":420,"thread_start_us":442,"threads_started":1}
I20260806 02:50:45.813129 21711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a3d69b27f22403bbc708ff5842b2878" candidate_uuid: "69b4df6ad612461b8fca75d7f0293a4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fa98bf3b2584963852d66c5bd93fea1" is_pre_election: true
I20260806 02:50:45.813135 21574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a3d69b27f22403bbc708ff5842b2878" candidate_uuid: "69b4df6ad612461b8fca75d7f0293a4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "329a80a10d7f4ee9ae05c2e18048c77d" is_pre_election: true
I20260806 02:50:45.813786 21711 raft_consensus.cc:2454] T 9a3d69b27f22403bbc708ff5842b2878 P 2fa98bf3b2584963852d66c5bd93fea1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69b4df6ad612461b8fca75d7f0293a4b in term 0.
I20260806 02:50:45.813786 21574 raft_consensus.cc:2454] T 9a3d69b27f22403bbc708ff5842b2878 P 329a80a10d7f4ee9ae05c2e18048c77d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69b4df6ad612461b8fca75d7f0293a4b in term 0.
I20260806 02:50:45.814752 21780 leader_election.cc:302] T 9a3d69b27f22403bbc708ff5842b2878 P 69b4df6ad612461b8fca75d7f0293a4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fa98bf3b2584963852d66c5bd93fea1, 69b4df6ad612461b8fca75d7f0293a4b; no voters:
I20260806 02:50:45.815279 21920 raft_consensus.cc:2781] T 9a3d69b27f22403bbc708ff5842b2878 P 69b4df6ad612461b8fca75d7f0293a4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:50:45.815451 21920 raft_consensus.cc:482] T 9a3d69b27f22403bbc708ff5842b2878 P 69b4df6ad612461b8fca75d7f0293a4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:50:45.815562 21920 raft_consensus.cc:3037] T 9a3d69b27f22403bbc708ff5842b2878 P 69b4df6ad612461b8fca75d7f0293a4b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:45.817970 21920 raft_consensus.cc:504] T 9a3d69b27f22403bbc708ff5842b2878 P 69b4df6ad612461b8fca75d7f0293a4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69b4df6ad612461b8fca75d7f0293a4b" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41485 } } peers { permanent_uuid: "2fa98bf3b2584963852d66c5bd93fea1" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 46771 } } peers { permanent_uuid: "329a80a10d7f4ee9ae05c2e18048c77d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 40561 } }
I20260806 02:50:45.818665 21920 leader_election.cc:288] T 9a3d69b27f22403bbc708ff5842b2878 P 69b4df6ad612461b8fca75d7f0293a4b [CANDIDATE]: Term 1 election: Requested vote from peers 2fa98bf3b2584963852d66c5bd93fea1 (127.1.210.66:46771), 329a80a10d7f4ee9ae05c2e18048c77d (127.1.210.65:40561)
I20260806 02:50:45.819419 21711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a3d69b27f22403bbc708ff5842b2878" candidate_uuid: "69b4df6ad612461b8fca75d7f0293a4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fa98bf3b2584963852d66c5bd93fea1"
I20260806 02:50:45.819430 21574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a3d69b27f22403bbc708ff5842b2878" candidate_uuid: "69b4df6ad612461b8fca75d7f0293a4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "329a80a10d7f4ee9ae05c2e18048c77d"
I20260806 02:50:45.819674 21711 raft_consensus.cc:3037] T 9a3d69b27f22403bbc708ff5842b2878 P 2fa98bf3b2584963852d66c5bd93fea1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:45.819685 21574 raft_consensus.cc:3037] T 9a3d69b27f22403bbc708ff5842b2878 P 329a80a10d7f4ee9ae05c2e18048c77d [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:45.822481 21574 raft_consensus.cc:2454] T 9a3d69b27f22403bbc708ff5842b2878 P 329a80a10d7f4ee9ae05c2e18048c77d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69b4df6ad612461b8fca75d7f0293a4b in term 1.
I20260806 02:50:45.822496 21711 raft_consensus.cc:2454] T 9a3d69b27f22403bbc708ff5842b2878 P 2fa98bf3b2584963852d66c5bd93fea1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69b4df6ad612461b8fca75d7f0293a4b in term 1.
I20260806 02:50:45.823141 21782 leader_election.cc:302] T 9a3d69b27f22403bbc708ff5842b2878 P 69b4df6ad612461b8fca75d7f0293a4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2fa98bf3b2584963852d66c5bd93fea1, 329a80a10d7f4ee9ae05c2e18048c77d, 69b4df6ad612461b8fca75d7f0293a4b; no voters:
I20260806 02:50:45.823582 21920 raft_consensus.cc:2781] T 9a3d69b27f22403bbc708ff5842b2878 P 69b4df6ad612461b8fca75d7f0293a4b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:45.824119 21920 raft_consensus.cc:686] T 9a3d69b27f22403bbc708ff5842b2878 P 69b4df6ad612461b8fca75d7f0293a4b [term 1 LEADER]: Becoming Leader. State: Replica: 69b4df6ad612461b8fca75d7f0293a4b, State: Running, Role: LEADER
I20260806 02:50:45.824622 21920 consensus_queue.cc:237] T 9a3d69b27f22403bbc708ff5842b2878 P 69b4df6ad612461b8fca75d7f0293a4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69b4df6ad612461b8fca75d7f0293a4b" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41485 } } peers { permanent_uuid: "2fa98bf3b2584963852d66c5bd93fea1" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 46771 } } peers { permanent_uuid: "329a80a10d7f4ee9ae05c2e18048c77d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 40561 } }
I20260806 02:50:45.830233 21427 catalog_manager.cc:5495] T 9a3d69b27f22403bbc708ff5842b2878 P 69b4df6ad612461b8fca75d7f0293a4b reported cstate change: term changed from 0 to 1, leader changed from <none> to 69b4df6ad612461b8fca75d7f0293a4b (127.1.210.67). New cstate: current_term: 1 leader_uuid: "69b4df6ad612461b8fca75d7f0293a4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69b4df6ad612461b8fca75d7f0293a4b" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 41485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fa98bf3b2584963852d66c5bd93fea1" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 46771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "329a80a10d7f4ee9ae05c2e18048c77d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20260806 02:50:45.850962 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:45.845113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40192 (local address 127.1.210.67:41485)
0806 02:50:45.845342 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:45.845347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:45.845365 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:50:45.845461 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:45.845465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:45.845552 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:45.845693 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:50:45.846108 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.846957 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.848455 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.848889 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.849007 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:45.849634 (+ 627us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:50:45.850221 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:50:45.850350 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:45.850535 (+ 185us) server_negotiation.cc:299] Negotiation successful
0806 02:50:45.850612 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260806 02:50:45.859926 21711 raft_consensus.cc:1261] T 9a3d69b27f22403bbc708ff5842b2878 P 2fa98bf3b2584963852d66c5bd93fea1 [term 1 FOLLOWER]: Refusing update from remote peer 69b4df6ad612461b8fca75d7f0293a4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:50:45.859925 21574 raft_consensus.cc:1261] T 9a3d69b27f22403bbc708ff5842b2878 P 329a80a10d7f4ee9ae05c2e18048c77d [term 1 FOLLOWER]: Refusing update from remote peer 69b4df6ad612461b8fca75d7f0293a4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:50:45.860942 21922 consensus_queue.cc:1035] T 9a3d69b27f22403bbc708ff5842b2878 P 69b4df6ad612461b8fca75d7f0293a4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fa98bf3b2584963852d66c5bd93fea1" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 46771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:50:45.861482 21920 consensus_queue.cc:1035] T 9a3d69b27f22403bbc708ff5842b2878 P 69b4df6ad612461b8fca75d7f0293a4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "329a80a10d7f4ee9ae05c2e18048c77d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 02:50:45.869589 21894 tablet.cc:2369] T 9a3d69b27f22403bbc708ff5842b2878 P 69b4df6ad612461b8fca75d7f0293a4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:50:45.875891 21925 mvcc.cc:205] Tried to move back new op lower bound from 7315393109431537664 to 7315393109301809152. Current Snapshot: MvccSnapshot[applied={T|T < 7315393109431537664}]
I20260806 02:50:45.895694 21426 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:50:45.896067 21426 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:50:45.900489 21426 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7adf156c12dc48febf4bd1ff70cd429b: Sending DeleteTablet for 3 replicas of tablet 9a3d69b27f22403bbc708ff5842b2878
I20260806 02:50:45.901757 21827 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a3d69b27f22403bbc708ff5842b2878 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:45 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:40184
I20260806 02:50:45.901784 21691 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a3d69b27f22403bbc708ff5842b2878 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:45 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:38814
I20260806 02:50:45.902827 21554 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a3d69b27f22403bbc708ff5842b2878 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:45 UTC) from {username='kudu', principal='kudu/127.1.210.126@KRBTEST.COM'} at 127.0.0.1:57006
I20260806 02:50:45.903306 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 21490
I20260806 02:50:45.914173 21937 tablet_replica.cc:330] T 9a3d69b27f22403bbc708ff5842b2878 P 329a80a10d7f4ee9ae05c2e18048c77d: stopping tablet replica
I20260806 02:50:45.914469 21936 tablet_replica.cc:330] T 9a3d69b27f22403bbc708ff5842b2878 P 2fa98bf3b2584963852d66c5bd93fea1: stopping tablet replica
I20260806 02:50:45.914834 21937 raft_consensus.cc:2229] T 9a3d69b27f22403bbc708ff5842b2878 P 329a80a10d7f4ee9ae05c2e18048c77d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:50:45.915165 21936 raft_consensus.cc:2229] T 9a3d69b27f22403bbc708ff5842b2878 P 2fa98bf3b2584963852d66c5bd93fea1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:50:45.915498 21937 raft_consensus.cc:2258] T 9a3d69b27f22403bbc708ff5842b2878 P 329a80a10d7f4ee9ae05c2e18048c77d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:45.915807 21936 raft_consensus.cc:2258] T 9a3d69b27f22403bbc708ff5842b2878 P 2fa98bf3b2584963852d66c5bd93fea1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:45.918031 21936 ts_tablet_manager.cc:1905] T 9a3d69b27f22403bbc708ff5842b2878 P 2fa98bf3b2584963852d66c5bd93fea1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:45.917999 21937 ts_tablet_manager.cc:1905] T 9a3d69b27f22403bbc708ff5842b2878 P 329a80a10d7f4ee9ae05c2e18048c77d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:45.925823 21936 ts_tablet_manager.cc:1918] T 9a3d69b27f22403bbc708ff5842b2878 P 2fa98bf3b2584963852d66c5bd93fea1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:50:45.925823 21937 ts_tablet_manager.cc:1918] T 9a3d69b27f22403bbc708ff5842b2878 P 329a80a10d7f4ee9ae05c2e18048c77d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:50:45.926056 21937 log.cc:1196] T 9a3d69b27f22403bbc708ff5842b2878 P 329a80a10d7f4ee9ae05c2e18048c77d: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-0/wal/wals/9a3d69b27f22403bbc708ff5842b2878
I20260806 02:50:45.926227 21936 log.cc:1196] T 9a3d69b27f22403bbc708ff5842b2878 P 2fa98bf3b2584963852d66c5bd93fea1: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-1/wal/wals/9a3d69b27f22403bbc708ff5842b2878
I20260806 02:50:45.926772 21937 ts_tablet_manager.cc:1939] T 9a3d69b27f22403bbc708ff5842b2878 P 329a80a10d7f4ee9ae05c2e18048c77d: Deleting consensus metadata
I20260806 02:50:45.926999 21936 ts_tablet_manager.cc:1939] T 9a3d69b27f22403bbc708ff5842b2878 P 2fa98bf3b2584963852d66c5bd93fea1: Deleting consensus metadata
I20260806 02:50:45.928906 21412 catalog_manager.cc:4841] TS 2fa98bf3b2584963852d66c5bd93fea1 (127.1.210.66:46771): tablet 9a3d69b27f22403bbc708ff5842b2878 (table test-table [id=cc875474ee70416e92262036d75fb387]) successfully deleted
I20260806 02:50:45.929051 21414 catalog_manager.cc:4841] TS 329a80a10d7f4ee9ae05c2e18048c77d (127.1.210.65:40561): tablet 9a3d69b27f22403bbc708ff5842b2878 (table test-table [id=cc875474ee70416e92262036d75fb387]) successfully deleted
I20260806 02:50:45.929945 21935 tablet_replica.cc:330] T 9a3d69b27f22403bbc708ff5842b2878 P 69b4df6ad612461b8fca75d7f0293a4b: stopping tablet replica
I20260806 02:50:45.930390 21935 raft_consensus.cc:2229] T 9a3d69b27f22403bbc708ff5842b2878 P 69b4df6ad612461b8fca75d7f0293a4b [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:50:45.931380 21935 raft_consensus.cc:2258] T 9a3d69b27f22403bbc708ff5842b2878 P 69b4df6ad612461b8fca75d7f0293a4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:45.934155 21935 ts_tablet_manager.cc:1905] T 9a3d69b27f22403bbc708ff5842b2878 P 69b4df6ad612461b8fca75d7f0293a4b: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 06 02:50:45 dist-test-slave-8tdl krb5kdc[21381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.65@KRBTEST.COM
I20260806 02:50:45.942293 21935 ts_tablet_manager.cc:1918] T 9a3d69b27f22403bbc708ff5842b2878 P 69b4df6ad612461b8fca75d7f0293a4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:50:45.942480 21935 log.cc:1196] T 9a3d69b27f22403bbc708ff5842b2878 P 69b4df6ad612461b8fca75d7f0293a4b: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984304136196-1865-0/minicluster-data/ts-2/wal/wals/9a3d69b27f22403bbc708ff5842b2878
I20260806 02:50:45.943145 21935 ts_tablet_manager.cc:1939] T 9a3d69b27f22403bbc708ff5842b2878 P 69b4df6ad612461b8fca75d7f0293a4b: Deleting consensus metadata
I20260806 02:50:45.947149 21911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:45.904806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57008 (local address 127.1.210.65:40561)
0806 02:50:45.904985 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:45.904992 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:45.928622 (+ 23630us) server_negotiation.cc:407] Connection header received
0806 02:50:45.929136 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:45.929143 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:45.929232 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:45.929317 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:45.931343 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.932569 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.934356 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:45.934917 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:45.935145 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:45.940006 (+ 4861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:45.940025 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:45.940040 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:45.940083 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:45.943957 (+ 3874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:45.944779 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:45.944787 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:45.944789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:45.944939 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:45.945364 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:45.945367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:45.945369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:45.945809 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:45.946159 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:45.946468 (+ 309us) server_negotiation.cc:299] Negotiation successful
0806 02:50:45.946732 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:50:45.947644 21413 catalog_manager.cc:4841] TS 69b4df6ad612461b8fca75d7f0293a4b (127.1.210.67:41485): tablet 9a3d69b27f22403bbc708ff5842b2878 (table test-table [id=cc875474ee70416e92262036d75fb387]) successfully deleted
I20260806 02:50:45.947938 21614 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:50:46.088646 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 21490
I20260806 02:50:46.110570 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 21625
Aug 06 02:50:46 dist-test-slave-8tdl krb5kdc[21381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.66@KRBTEST.COM
I20260806 02:50:46.130257 21909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:46.111734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38818 (local address 127.1.210.66:46771)
0806 02:50:46.111898 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:46.111903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:46.111924 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:50:46.111998 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:46.112002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:46.112068 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:46.112160 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:46.114061 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:46.115110 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:46.117163 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:46.117601 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:46.117713 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:46.123491 (+ 5778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:46.123517 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:46.123540 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:46.123598 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:46.127105 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:46.127976 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:46.127984 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:46.127987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:46.128136 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:46.128562 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:46.128568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:46.128573 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:46.129006 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:46.129319 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:46.129643 (+ 324us) server_negotiation.cc:299] Negotiation successful
0806 02:50:46.129878 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 02:50:46.131227 21752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:50:46.224902 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 21625
I20260806 02:50:46.244488 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 21763
Aug 06 02:50:46 dist-test-slave-8tdl krb5kdc[21381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.67@KRBTEST.COM
I20260806 02:50:46.262342 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:46.245399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40204 (local address 127.1.210.67:41485)
0806 02:50:46.245579 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:46.245587 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:46.245682 (+ 95us) server_negotiation.cc:407] Connection header received
0806 02:50:46.245893 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:46.245897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:46.245964 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:46.246093 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:46.247462 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:46.248354 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:46.249700 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:46.250187 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:46.250335 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:46.255453 (+ 5118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:46.255468 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:46.255488 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:46.255570 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:46.258956 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:46.259800 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:46.259814 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:46.259816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:46.259986 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:46.260400 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:46.260406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:46.260409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:46.260844 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:46.261196 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:46.261597 (+ 401us) server_negotiation.cc:299] Negotiation successful
0806 02:50:46.261895 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 02:50:46.263257 21888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:50:46.392953 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 21763
I20260806 02:50:46.415547 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 21394
I20260806 02:50:46.416754 21458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:50:46.535908 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 21394
2026-08-06T02:50:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4909 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-06T02:50:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:46Z Disabled control of system clock
I20260806 02:50:46.632237 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:35957
--webserver_interface=127.1.210.126
--webserver_port=0
--builtin_ntp_servers=127.1.210.84:41573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:35957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 02:50:46.996641 21946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:46.996961 21946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:46.997046 21946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:47.004900 21946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:47.005112 21946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:47.005167 21946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:47.005235 21946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:47.005290 21946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:47.015625 21946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:41573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:35957
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:35957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:50:47.017043 21946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:47.018909 21946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:47.032269 21951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:47.032194 21954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:47.032297 21952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:47.033944 21946 server_base.cc:1034] running on GCE node
I20260806 02:50:47.035041 21946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:47.036756 21946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:47.038045 21946 hybrid_clock.cc:648] HybridClock initialized: now 1785984647037944 us; error 53 us; skew 500 ppm
I20260806 02:50:47.038511 21946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:50:47.045280 21946 webserver.cc:460] Webserver started at http://127.1.210.126:35033/ using document root <none> and password file <none>
I20260806 02:50:47.046137 21946 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:47.046247 21946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:47.046639 21946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:47.050346 21946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "108a20824fa64ae3b989a1dfcb1cd04f"
format_stamp: "Formatted at 2026-08-06 02:50:47 on dist-test-slave-8tdl"
server_key: "86e9e43e5c57444ef64afdb3ab60d733"
server_key_iv: "3ddf8ccce185d20ca93e489a109fef70"
server_key_version: "encryptionkey@0"
I20260806 02:50:47.051407 21946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "108a20824fa64ae3b989a1dfcb1cd04f"
format_stamp: "Formatted at 2026-08-06 02:50:47 on dist-test-slave-8tdl"
server_key: "86e9e43e5c57444ef64afdb3ab60d733"
server_key_iv: "3ddf8ccce185d20ca93e489a109fef70"
server_key_version: "encryptionkey@0"
I20260806 02:50:47.057787 21946 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 02:50:47.062367 21961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:47.064352 21946 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:50:47.064615 21946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "108a20824fa64ae3b989a1dfcb1cd04f"
format_stamp: "Formatted at 2026-08-06 02:50:47 on dist-test-slave-8tdl"
server_key: "86e9e43e5c57444ef64afdb3ab60d733"
server_key_iv: "3ddf8ccce185d20ca93e489a109fef70"
server_key_version: "encryptionkey@0"
I20260806 02:50:47.064890 21946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:47.091930 21946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:47.093063 21946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:47.093398 21946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:47.116647 21946 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:35957
I20260806 02:50:47.116650 22013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:35957 every 8 connection(s)
I20260806 02:50:47.118281 21946 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:47.122540 22014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:47.124811 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 21946
I20260806 02:50:47.125026 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:50:47.125594 1865 external_mini_cluster.cc:1428] Setting key acc3ce14767d6e64dc60d799814afd19
I20260806 02:50:47.136538 22014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 108a20824fa64ae3b989a1dfcb1cd04f: Bootstrap starting.
I20260806 02:50:47.140777 22014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 108a20824fa64ae3b989a1dfcb1cd04f: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:47.142102 22014 log.cc:824] T 00000000000000000000000000000000 P 108a20824fa64ae3b989a1dfcb1cd04f: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:47.143244 22017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:47.127919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56364 (local address 127.1.210.126:35957)
0806 02:50:47.129227 (+ 1308us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:47.129240 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:47.129290 (+ 50us) server_negotiation.cc:407] Connection header received
0806 02:50:47.131411 (+ 2121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:47.131440 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:47.132055 (+ 615us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:47.132479 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:47.133271 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:47.134912 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:47.136930 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:47.137820 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:47.137995 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:47.139691 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:47.139700 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:47.139717 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:47.140402 (+ 685us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:47.140520 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:47.140546 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:47.140668 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:47.141049 (+ 381us) server_negotiation.cc:299] Negotiation successful
0806 02:50:47.141543 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":766,"thread_start_us":480,"threads_started":1}
I20260806 02:50:47.145597 22014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 108a20824fa64ae3b989a1dfcb1cd04f: No bootstrap required, opened a new log
I20260806 02:50:47.154546 22014 raft_consensus.cc:348] T 00000000000000000000000000000000 P 108a20824fa64ae3b989a1dfcb1cd04f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "108a20824fa64ae3b989a1dfcb1cd04f" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 35957 } }
I20260806 02:50:47.155121 22014 raft_consensus.cc:374] T 00000000000000000000000000000000 P 108a20824fa64ae3b989a1dfcb1cd04f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:47.155242 22014 raft_consensus.cc:729] T 00000000000000000000000000000000 P 108a20824fa64ae3b989a1dfcb1cd04f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 108a20824fa64ae3b989a1dfcb1cd04f, State: Initialized, Role: FOLLOWER
I20260806 02:50:47.155906 22014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 108a20824fa64ae3b989a1dfcb1cd04f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "108a20824fa64ae3b989a1dfcb1cd04f" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 35957 } }
I20260806 02:50:47.156200 22014 raft_consensus.cc:388] T 00000000000000000000000000000000 P 108a20824fa64ae3b989a1dfcb1cd04f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:50:47.156379 22014 raft_consensus.cc:482] T 00000000000000000000000000000000 P 108a20824fa64ae3b989a1dfcb1cd04f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:50:47.156576 22014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 108a20824fa64ae3b989a1dfcb1cd04f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:47.159018 22014 raft_consensus.cc:504] T 00000000000000000000000000000000 P 108a20824fa64ae3b989a1dfcb1cd04f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "108a20824fa64ae3b989a1dfcb1cd04f" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 35957 } }
I20260806 02:50:47.159621 22014 leader_election.cc:302] T 00000000000000000000000000000000 P 108a20824fa64ae3b989a1dfcb1cd04f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 108a20824fa64ae3b989a1dfcb1cd04f; no voters:
I20260806 02:50:47.160316 22014 leader_election.cc:288] T 00000000000000000000000000000000 P 108a20824fa64ae3b989a1dfcb1cd04f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:50:47.160589 22019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 108a20824fa64ae3b989a1dfcb1cd04f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:47.161478 22019 raft_consensus.cc:686] T 00000000000000000000000000000000 P 108a20824fa64ae3b989a1dfcb1cd04f [term 1 LEADER]: Becoming Leader. State: Replica: 108a20824fa64ae3b989a1dfcb1cd04f, State: Running, Role: LEADER
I20260806 02:50:47.162150 22019 consensus_queue.cc:237] T 00000000000000000000000000000000 P 108a20824fa64ae3b989a1dfcb1cd04f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "108a20824fa64ae3b989a1dfcb1cd04f" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 35957 } }
I20260806 02:50:47.162583 22014 sys_catalog.cc:564] T 00000000000000000000000000000000 P 108a20824fa64ae3b989a1dfcb1cd04f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:47.166290 22020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 108a20824fa64ae3b989a1dfcb1cd04f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "108a20824fa64ae3b989a1dfcb1cd04f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "108a20824fa64ae3b989a1dfcb1cd04f" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 35957 } } }
I20260806 02:50:47.166953 22020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 108a20824fa64ae3b989a1dfcb1cd04f [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:47.166290 22021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 108a20824fa64ae3b989a1dfcb1cd04f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 108a20824fa64ae3b989a1dfcb1cd04f. Latest consensus state: current_term: 1 leader_uuid: "108a20824fa64ae3b989a1dfcb1cd04f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "108a20824fa64ae3b989a1dfcb1cd04f" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 35957 } } }
I20260806 02:50:47.167096 22021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 108a20824fa64ae3b989a1dfcb1cd04f [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:47.169180 22027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:50:47.175972 22027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:47.187232 22027 catalog_manager.cc:1347] Generated new cluster ID: 0e534cdd2f594acc941df3cdf37fc852
I20260806 02:50:47.187427 22027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:47.209501 22027 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:50:47.211355 22027 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:47.219291 22027 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 108a20824fa64ae3b989a1dfcb1cd04f: Generated new TSK 0
I20260806 02:50:47.220504 22027 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:50:47.236991 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.65:0
--local_ip_for_outbound_sockets=127.1.210.65
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:35957
--builtin_ntp_servers=127.1.210.84:41573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 02:50:47.618468 22038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:47.618858 22038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:47.619032 22038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:47.627538 22038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:47.627640 22038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:47.627827 22038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.65
I20260806 02:50:47.637701 22038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:41573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.210.65
--webserver_port=0
--tserver_master_addrs=127.1.210.126:35957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.65
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:50:47.639425 22038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:47.641420 22038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:47.655788 22043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:47.655893 22044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:47.655833 22046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:47.662523 22038 server_base.cc:1034] running on GCE node
I20260806 02:50:47.663357 22038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:47.664647 22038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:47.665880 22038 hybrid_clock.cc:648] HybridClock initialized: now 1785984647665833 us; error 39 us; skew 500 ppm
I20260806 02:50:47.666414 22038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:50:47.669587 22038 webserver.cc:460] Webserver started at http://127.1.210.65:46039/ using document root <none> and password file <none>
I20260806 02:50:47.670372 22038 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:47.670470 22038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:47.670837 22038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:47.674190 22038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-0/data/instance:
uuid: "4bc8486434d34ebbaa88618f3ce7655d"
format_stamp: "Formatted at 2026-08-06 02:50:47 on dist-test-slave-8tdl"
server_key: "327d08ad3d18b397773331dfca87a509"
server_key_iv: "f67416f04490328eb049198f986fb122"
server_key_version: "encryptionkey@0"
I20260806 02:50:47.675127 22038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance:
uuid: "4bc8486434d34ebbaa88618f3ce7655d"
format_stamp: "Formatted at 2026-08-06 02:50:47 on dist-test-slave-8tdl"
server_key: "327d08ad3d18b397773331dfca87a509"
server_key_iv: "f67416f04490328eb049198f986fb122"
server_key_version: "encryptionkey@0"
I20260806 02:50:47.680783 22038 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 02:50:47.684988 22052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:47.686776 22038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:50:47.687001 22038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-0/wal
uuid: "4bc8486434d34ebbaa88618f3ce7655d"
format_stamp: "Formatted at 2026-08-06 02:50:47 on dist-test-slave-8tdl"
server_key: "327d08ad3d18b397773331dfca87a509"
server_key_iv: "f67416f04490328eb049198f986fb122"
server_key_version: "encryptionkey@0"
I20260806 02:50:47.687274 22038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:47.712193 22038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:47.713138 22038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:47.713490 22038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:47.714605 22038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:47.716348 22038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:47.716495 22038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:47.716627 22038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:47.716715 22038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:47.763443 22038 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.65:46593
I20260806 02:50:47.763548 22166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.65:46593 every 8 connection(s)
I20260806 02:50:47.764942 22038 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-0/data/info.pb
I20260806 02:50:47.768733 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 22038
I20260806 02:50:47.768946 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-0/wal/instance
I20260806 02:50:47.769618 1865 external_mini_cluster.cc:1428] Setting key 18572287173299bd5d191bf5e0ad8f23
I20260806 02:50:47.772848 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.66:0
--local_ip_for_outbound_sockets=127.1.210.66
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:35957
--builtin_ntp_servers=127.1.210.84:41573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 02:50:47.781757 22169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:47.768070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.65:45055 (local address 127.1.210.126:35957)
0806 02:50:47.768674 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:47.768681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:47.770359 (+ 1678us) server_negotiation.cc:407] Connection header received
0806 02:50:47.772710 (+ 2351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:47.772719 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:47.772816 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:47.773019 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:47.774507 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:47.775490 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:47.778174 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:47.778671 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:47.778883 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:47.780796 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:47.780801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:47.780805 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:47.780827 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:47.780884 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:47.780897 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:47.781016 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:47.781343 (+ 327us) server_negotiation.cc:299] Negotiation successful
0806 02:50:47.781452 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":298,"threads_started":1}
I20260806 02:50:47.783623 22170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:47.768830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:35957 (local address 127.1.210.65:45055)
0806 02:50:47.770162 (+ 1332us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:47.770250 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:47.772292 (+ 2042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:47.773226 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:47.773241 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:47.773380 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:47.774293 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:47.774323 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:47.775694 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:47.775703 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:47.777963 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:47.777984 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:47.779022 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:47.779032 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:47.779417 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:47.780337 (+ 920us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:50:47.780343 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:47.780404 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:50:47.780537 (+ 133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:47.781044 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:47.781059 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:47.781071 (+ 12us) client_negotiation.cc:769] Sending connection context
0806 02:50:47.781290 (+ 219us) client_negotiation.cc:240] Negotiation successful
0806 02:50:47.781791 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":317,"threads_started":1}
I20260806 02:50:47.785585 22167 heartbeater.cc:347] Connected to a master server at 127.1.210.126:35957
I20260806 02:50:47.786046 22167 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:47.787050 22167 heartbeater.cc:511] Master 127.1.210.126:35957 requested a full tablet report, sending...
I20260806 02:50:47.789721 21978 ts_manager.cc:194] Registered new tserver with Master: 4bc8486434d34ebbaa88618f3ce7655d (127.1.210.65:46593)
I20260806 02:50:47.792356 21978 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.210.65:45055
W20260806 02:50:48.140410 22171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:48.140718 22171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:48.140808 22171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:48.148383 22171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:48.148500 22171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:48.148679 22171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.66
I20260806 02:50:48.158947 22171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:41573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.210.66
--webserver_port=0
--tserver_master_addrs=127.1.210.126:35957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.66
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:50:48.160362 22171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:48.162286 22171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:48.176775 22176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:48.176746 22177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:48.176689 22179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:48.178328 22171 server_base.cc:1034] running on GCE node
I20260806 02:50:48.179149 22171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:48.180291 22171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:48.181533 22171 hybrid_clock.cc:648] HybridClock initialized: now 1785984648181449 us; error 63 us; skew 500 ppm
I20260806 02:50:48.181844 22171 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:50:48.184491 22171 webserver.cc:460] Webserver started at http://127.1.210.66:39731/ using document root <none> and password file <none>
I20260806 02:50:48.185232 22171 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:48.185325 22171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:48.185617 22171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:48.189150 22171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-1/data/instance:
uuid: "511175ba711842d0938410818d4e01d5"
format_stamp: "Formatted at 2026-08-06 02:50:48 on dist-test-slave-8tdl"
server_key: "659d8210876f32b9ed099a440af82276"
server_key_iv: "550a6738c006249a87bcf7d51f342abc"
server_key_version: "encryptionkey@0"
I20260806 02:50:48.190026 22171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance:
uuid: "511175ba711842d0938410818d4e01d5"
format_stamp: "Formatted at 2026-08-06 02:50:48 on dist-test-slave-8tdl"
server_key: "659d8210876f32b9ed099a440af82276"
server_key_iv: "550a6738c006249a87bcf7d51f342abc"
server_key_version: "encryptionkey@0"
I20260806 02:50:48.196314 22171 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260806 02:50:48.201299 22186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:48.203270 22171 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 02:50:48.203459 22171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-1/wal
uuid: "511175ba711842d0938410818d4e01d5"
format_stamp: "Formatted at 2026-08-06 02:50:48 on dist-test-slave-8tdl"
server_key: "659d8210876f32b9ed099a440af82276"
server_key_iv: "550a6738c006249a87bcf7d51f342abc"
server_key_version: "encryptionkey@0"
I20260806 02:50:48.203689 22171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:48.223301 22171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:48.224287 22171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:48.224610 22171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:48.225714 22171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:48.227474 22171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:48.227586 22171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:48.227677 22171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:48.227744 22171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:48.265277 22171 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.66:33707
I20260806 02:50:48.265395 22301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.66:33707 every 8 connection(s)
I20260806 02:50:48.266803 22171 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-1/data/info.pb
I20260806 02:50:48.271145 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 22171
I20260806 02:50:48.271345 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-1/wal/instance
I20260806 02:50:48.271947 1865 external_mini_cluster.cc:1428] Setting key 4fb7a83aad451893c723b06e20d2085c
I20260806 02:50:48.275108 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.210.67:0
--local_ip_for_outbound_sockets=127.1.210.67
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:35957
--builtin_ntp_servers=127.1.210.84:41573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 02:50:48.281965 22169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:48.269748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.66:36535 (local address 127.1.210.126:35957)
0806 02:50:48.269944 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:48.269949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:48.271583 (+ 1634us) server_negotiation.cc:407] Connection header received
0806 02:50:48.274054 (+ 2471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:48.274060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:48.274150 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:48.274260 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:48.275615 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.276861 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.278572 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.279309 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.279455 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:48.280914 (+ 1459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:48.280931 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:48.280939 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:48.280969 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:48.281016 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:48.281035 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:48.281132 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:48.281362 (+ 230us) server_negotiation.cc:299] Negotiation successful
0806 02:50:48.281539 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 02:50:48.283010 22304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:48.270306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:35957 (local address 127.1.210.66:36535)
0806 02:50:48.271372 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:48.271445 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:48.273690 (+ 2245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:48.274490 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:48.274499 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:48.274604 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:48.275432 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:48.275454 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.277063 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.277067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:48.278406 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:48.278420 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.279530 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.279535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:48.279759 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:48.280538 (+ 779us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:50:48.280541 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:48.280595 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:50:48.280687 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:48.281140 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:48.281150 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:48.281156 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 02:50:48.281318 (+ 162us) client_negotiation.cc:240] Negotiation successful
0806 02:50:48.281666 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":351,"threads_started":1}
I20260806 02:50:48.284577 22302 heartbeater.cc:347] Connected to a master server at 127.1.210.126:35957
I20260806 02:50:48.284914 22302 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:48.285743 22302 heartbeater.cc:511] Master 127.1.210.126:35957 requested a full tablet report, sending...
I20260806 02:50:48.287304 21978 ts_manager.cc:194] Registered new tserver with Master: 511175ba711842d0938410818d4e01d5 (127.1.210.66:33707)
I20260806 02:50:48.288534 21978 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.210.66:36535
W20260806 02:50:48.633333 22305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:48.633635 22305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:48.633796 22305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:48.641583 22305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:48.641703 22305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:48.641892 22305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.210.67
I20260806 02:50:48.651392 22305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:41573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.210.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.210.67
--webserver_port=0
--tserver_master_addrs=127.1.210.126:35957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.210.67
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:50:48.652765 22305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:48.654641 22305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:48.668612 22310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:48.668511 22313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:48.668653 22311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:48.672490 22305 server_base.cc:1034] running on GCE node
I20260806 02:50:48.673185 22305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:48.674256 22305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:48.675508 22305 hybrid_clock.cc:648] HybridClock initialized: now 1785984648675441 us; error 64 us; skew 500 ppm
I20260806 02:50:48.675873 22305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:50:48.678351 22305 webserver.cc:460] Webserver started at http://127.1.210.67:34935/ using document root <none> and password file <none>
I20260806 02:50:48.679199 22305 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:48.679339 22305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:48.679692 22305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:48.682747 22305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-2/data/instance:
uuid: "f1664d39ba29483dbded606a5ed784af"
format_stamp: "Formatted at 2026-08-06 02:50:48 on dist-test-slave-8tdl"
server_key: "8a23bfc1ebc3780dd415b0eda3691bb3"
server_key_iv: "58c830071ad1834d678c2f46708486e8"
server_key_version: "encryptionkey@0"
I20260806 02:50:48.683629 22305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance:
uuid: "f1664d39ba29483dbded606a5ed784af"
format_stamp: "Formatted at 2026-08-06 02:50:48 on dist-test-slave-8tdl"
server_key: "8a23bfc1ebc3780dd415b0eda3691bb3"
server_key_iv: "58c830071ad1834d678c2f46708486e8"
server_key_version: "encryptionkey@0"
I20260806 02:50:48.689246 22305 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260806 02:50:48.693190 22319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:48.694615 22305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:50:48.694818 22305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-2/wal
uuid: "f1664d39ba29483dbded606a5ed784af"
format_stamp: "Formatted at 2026-08-06 02:50:48 on dist-test-slave-8tdl"
server_key: "8a23bfc1ebc3780dd415b0eda3691bb3"
server_key_iv: "58c830071ad1834d678c2f46708486e8"
server_key_version: "encryptionkey@0"
I20260806 02:50:48.695076 22305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:48.738744 22305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:48.739696 22305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:48.740046 22305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:48.741520 22305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:50:48.743271 22305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:50:48.743366 22305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:48.743508 22305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:50:48.743598 22305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:48.785378 22305 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.67:42977
I20260806 02:50:48.785435 22433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.67:42977 every 8 connection(s)
I20260806 02:50:48.786916 22305 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-2/data/info.pb
I20260806 02:50:48.791291 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 22305
I20260806 02:50:48.791443 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-2/wal/instance
I20260806 02:50:48.791927 1865 external_mini_cluster.cc:1428] Setting key a00995ebc1e95227fe3f9ac789433199
I20260806 02:50:48.796025 22167 heartbeater.cc:503] Master 127.1.210.126:35957 was elected leader, sending a full tablet report...
I20260806 02:50:48.801400 22436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:48.789877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:60839 (local address 127.1.210.126:35957)
0806 02:50:48.790333 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:48.790337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:48.791939 (+ 1602us) server_negotiation.cc:407] Connection header received
0806 02:50:48.794150 (+ 2211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:48.794155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:48.794253 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:48.794373 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:48.795597 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.796490 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.798163 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.798573 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.798748 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:48.800362 (+ 1614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:48.800367 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:48.800370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:48.800390 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:48.800430 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:48.800442 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:48.800512 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:48.800839 (+ 327us) server_negotiation.cc:299] Negotiation successful
0806 02:50:48.800934 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":234,"threads_started":1}
I20260806 02:50:48.802387 22437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:48.790562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:35957 (local address 127.1.210.67:60839)
0806 02:50:48.791748 (+ 1186us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:48.791796 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:48.793848 (+ 2052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:48.794633 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:48.794642 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:48.794799 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:48.795420 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:48.795437 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.796669 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.796673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:48.798002 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:48.798015 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.798852 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.798860 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:48.799183 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:48.800018 (+ 835us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:50:48.800021 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:48.800074 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:50:48.800164 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:48.800563 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:48.800572 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:48.800579 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 02:50:48.800760 (+ 181us) client_negotiation.cc:240] Negotiation successful
0806 02:50:48.801182 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":379,"threads_started":1}
I20260806 02:50:48.804296 22434 heartbeater.cc:347] Connected to a master server at 127.1.210.126:35957
I20260806 02:50:48.804639 22434 heartbeater.cc:464] Registering TS with master...
I20260806 02:50:48.805840 22434 heartbeater.cc:511] Master 127.1.210.126:35957 requested a full tablet report, sending...
I20260806 02:50:48.807854 21977 ts_manager.cc:194] Registered new tserver with Master: f1664d39ba29483dbded606a5ed784af (127.1.210.67:42977)
I20260806 02:50:48.809396 21977 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.210.67:60839
I20260806 02:50:48.821300 1865 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:50:48.835045 22436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:48.826793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56370 (local address 127.1.210.126:35957)
0806 02:50:48.827039 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:48.827047 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:48.827590 (+ 543us) server_negotiation.cc:407] Connection header received
0806 02:50:48.827812 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:48.827818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:48.827955 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:48.828057 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:48.828569 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.829472 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.831878 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.832462 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.832606 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:48.833849 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:48.833860 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:48.833869 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:48.833898 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:48.833957 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:48.833976 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:48.834093 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:48.834354 (+ 261us) server_negotiation.cc:299] Negotiation successful
0806 02:50:48.834490 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260806 02:50:48.841399 21977 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:50:48.854926 21977 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:50:48.885259 22447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:48.870955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:46593 (local address 127.0.0.1:43620)
0806 02:50:48.874467 (+ 3512us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:48.874523 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:48.874791 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:48.876055 (+ 1264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:48.876063 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:48.876104 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:48.876502 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:48.876519 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.878514 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.878522 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:48.881459 (+ 2937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:48.881477 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.884226 (+ 2749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.884232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:48.884423 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:48.884467 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 02:50:48.884627 (+ 160us) client_negotiation.cc:240] Negotiation successful
0806 02:50:48.884856 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3141,"thread_start_us":2940,"threads_started":1}
I20260806 02:50:48.885756 22450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:48.871686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43620 (local address 127.1.210.65:46593)
0806 02:50:48.873305 (+ 1619us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:48.873321 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:48.875074 (+ 1753us) server_negotiation.cc:407] Connection header received
0806 02:50:48.875192 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:48.875201 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:48.875495 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:48.875723 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:48.876806 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.878308 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.881739 (+ 3431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.883555 (+ 1816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.883701 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:48.884055 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:48.884740 (+ 685us) server_negotiation.cc:299] Negotiation successful
0806 02:50:48.884957 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1380,"thread_start_us":1237,"threads_started":1}
I20260806 02:50:48.886451 22448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:48.871091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.67:42977 (local address 127.0.0.1:39496)
0806 02:50:48.875333 (+ 4242us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:48.875355 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:48.875545 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:48.876521 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:48.876525 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:48.876546 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:48.877106 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:48.877120 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.879740 (+ 2620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.879747 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:48.882228 (+ 2481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:48.882245 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.885389 (+ 3144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.885410 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:48.885691 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:48.885748 (+ 57us) client_negotiation.cc:769] Sending connection context
0806 02:50:48.885865 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 02:50:48.886061 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4103,"thread_start_us":3929,"threads_started":1}
I20260806 02:50:48.887226 22449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:48.871227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39496 (local address 127.1.210.67:42977)
0806 02:50:48.874837 (+ 3610us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:48.874847 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:48.875491 (+ 644us) server_negotiation.cc:407] Connection header received
0806 02:50:48.875740 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:48.875747 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:48.875983 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:48.876201 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:48.877350 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.879545 (+ 2195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.882461 (+ 2916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.884654 (+ 2193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.884835 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:48.885201 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:48.886756 (+ 1555us) server_negotiation.cc:299] Negotiation successful
0806 02:50:48.886905 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2502,"thread_start_us":2286,"threads_started":1}
I20260806 02:50:48.890554 22100 tablet_service.cc:1467] Processing CreateTablet for tablet 8ece3179d99b427aa83536200cc1872f (DEFAULT_TABLE table=test-table [id=49c0f319f15e48819aedf8d14ee8a5fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:48.890862 22368 tablet_service.cc:1467] Processing CreateTablet for tablet 8ece3179d99b427aa83536200cc1872f (DEFAULT_TABLE table=test-table [id=49c0f319f15e48819aedf8d14ee8a5fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:48.892200 22100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ece3179d99b427aa83536200cc1872f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:48.892638 22368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ece3179d99b427aa83536200cc1872f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:48.895831 22447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:48.876370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:33707 (local address 127.0.0.1:58142)
0806 02:50:48.885939 (+ 9569us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:48.885961 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:48.887271 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:48.888452 (+ 1181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:48.888459 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:48.888492 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:48.888662 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:48.888673 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.890672 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.890705 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:48.892780 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:48.892800 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.894994 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.895001 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:48.895209 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:48.895250 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 02:50:48.895341 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 02:50:48.895517 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9407,"thread_start_us":10017,"threads_started":1}
I20260806 02:50:48.896437 22451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:48.875703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58142 (local address 127.1.210.66:33707)
0806 02:50:48.876361 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:48.876370 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:48.886078 (+ 9708us) server_negotiation.cc:407] Connection header received
0806 02:50:48.887568 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:48.887576 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:48.887811 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:48.888028 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:48.888919 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.890483 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.892966 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.894426 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.894578 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:48.894880 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:48.895800 (+ 920us) server_negotiation.cc:299] Negotiation successful
0806 02:50:48.896024 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":324,"threads_started":1}
I20260806 02:50:48.901366 22235 tablet_service.cc:1467] Processing CreateTablet for tablet 8ece3179d99b427aa83536200cc1872f (DEFAULT_TABLE table=test-table [id=49c0f319f15e48819aedf8d14ee8a5fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:50:48.903180 22235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ece3179d99b427aa83536200cc1872f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:48.906589 22453 tablet_bootstrap.cc:492] T 8ece3179d99b427aa83536200cc1872f P 4bc8486434d34ebbaa88618f3ce7655d: Bootstrap starting.
I20260806 02:50:48.910312 22453 tablet_bootstrap.cc:654] T 8ece3179d99b427aa83536200cc1872f P 4bc8486434d34ebbaa88618f3ce7655d: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:48.911381 22454 tablet_bootstrap.cc:492] T 8ece3179d99b427aa83536200cc1872f P f1664d39ba29483dbded606a5ed784af: Bootstrap starting.
I20260806 02:50:48.913693 22453 log.cc:824] T 8ece3179d99b427aa83536200cc1872f P 4bc8486434d34ebbaa88618f3ce7655d: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:48.915582 22454 tablet_bootstrap.cc:654] T 8ece3179d99b427aa83536200cc1872f P f1664d39ba29483dbded606a5ed784af: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:48.916878 22454 log.cc:824] T 8ece3179d99b427aa83536200cc1872f P f1664d39ba29483dbded606a5ed784af: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:48.917335 22453 tablet_bootstrap.cc:492] T 8ece3179d99b427aa83536200cc1872f P 4bc8486434d34ebbaa88618f3ce7655d: No bootstrap required, opened a new log
I20260806 02:50:48.917856 22453 ts_tablet_manager.cc:1397] T 8ece3179d99b427aa83536200cc1872f P 4bc8486434d34ebbaa88618f3ce7655d: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260806 02:50:48.919488 22458 tablet_bootstrap.cc:492] T 8ece3179d99b427aa83536200cc1872f P 511175ba711842d0938410818d4e01d5: Bootstrap starting.
I20260806 02:50:48.923504 22458 tablet_bootstrap.cc:654] T 8ece3179d99b427aa83536200cc1872f P 511175ba711842d0938410818d4e01d5: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:48.923835 22454 tablet_bootstrap.cc:492] T 8ece3179d99b427aa83536200cc1872f P f1664d39ba29483dbded606a5ed784af: No bootstrap required, opened a new log
I20260806 02:50:48.924170 22454 ts_tablet_manager.cc:1397] T 8ece3179d99b427aa83536200cc1872f P f1664d39ba29483dbded606a5ed784af: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.003s
I20260806 02:50:48.924755 22458 log.cc:824] T 8ece3179d99b427aa83536200cc1872f P 511175ba711842d0938410818d4e01d5: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:48.927589 22458 tablet_bootstrap.cc:492] T 8ece3179d99b427aa83536200cc1872f P 511175ba711842d0938410818d4e01d5: No bootstrap required, opened a new log
I20260806 02:50:48.927879 22458 ts_tablet_manager.cc:1397] T 8ece3179d99b427aa83536200cc1872f P 511175ba711842d0938410818d4e01d5: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260806 02:50:48.931089 22453 raft_consensus.cc:348] T 8ece3179d99b427aa83536200cc1872f P 4bc8486434d34ebbaa88618f3ce7655d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1664d39ba29483dbded606a5ed784af" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42977 } } peers { permanent_uuid: "4bc8486434d34ebbaa88618f3ce7655d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46593 } } peers { permanent_uuid: "511175ba711842d0938410818d4e01d5" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 33707 } }
I20260806 02:50:48.931646 22453 raft_consensus.cc:374] T 8ece3179d99b427aa83536200cc1872f P 4bc8486434d34ebbaa88618f3ce7655d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:48.931766 22453 raft_consensus.cc:729] T 8ece3179d99b427aa83536200cc1872f P 4bc8486434d34ebbaa88618f3ce7655d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc8486434d34ebbaa88618f3ce7655d, State: Initialized, Role: FOLLOWER
I20260806 02:50:48.932662 22453 consensus_queue.cc:260] T 8ece3179d99b427aa83536200cc1872f P 4bc8486434d34ebbaa88618f3ce7655d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1664d39ba29483dbded606a5ed784af" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42977 } } peers { permanent_uuid: "4bc8486434d34ebbaa88618f3ce7655d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46593 } } peers { permanent_uuid: "511175ba711842d0938410818d4e01d5" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 33707 } }
I20260806 02:50:48.934767 22453 ts_tablet_manager.cc:1428] T 8ece3179d99b427aa83536200cc1872f P 4bc8486434d34ebbaa88618f3ce7655d: Time spent starting tablet: real 0.017s user 0.011s sys 0.008s
I20260806 02:50:48.937402 22458 raft_consensus.cc:348] T 8ece3179d99b427aa83536200cc1872f P 511175ba711842d0938410818d4e01d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1664d39ba29483dbded606a5ed784af" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42977 } } peers { permanent_uuid: "4bc8486434d34ebbaa88618f3ce7655d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46593 } } peers { permanent_uuid: "511175ba711842d0938410818d4e01d5" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 33707 } }
I20260806 02:50:48.937402 22454 raft_consensus.cc:348] T 8ece3179d99b427aa83536200cc1872f P f1664d39ba29483dbded606a5ed784af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1664d39ba29483dbded606a5ed784af" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42977 } } peers { permanent_uuid: "4bc8486434d34ebbaa88618f3ce7655d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46593 } } peers { permanent_uuid: "511175ba711842d0938410818d4e01d5" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 33707 } }
I20260806 02:50:48.937947 22458 raft_consensus.cc:374] T 8ece3179d99b427aa83536200cc1872f P 511175ba711842d0938410818d4e01d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:48.938090 22454 raft_consensus.cc:374] T 8ece3179d99b427aa83536200cc1872f P f1664d39ba29483dbded606a5ed784af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:48.938088 22458 raft_consensus.cc:729] T 8ece3179d99b427aa83536200cc1872f P 511175ba711842d0938410818d4e01d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 511175ba711842d0938410818d4e01d5, State: Initialized, Role: FOLLOWER
I20260806 02:50:48.938265 22454 raft_consensus.cc:729] T 8ece3179d99b427aa83536200cc1872f P f1664d39ba29483dbded606a5ed784af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1664d39ba29483dbded606a5ed784af, State: Initialized, Role: FOLLOWER
I20260806 02:50:48.938927 22454 consensus_queue.cc:260] T 8ece3179d99b427aa83536200cc1872f P f1664d39ba29483dbded606a5ed784af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1664d39ba29483dbded606a5ed784af" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42977 } } peers { permanent_uuid: "4bc8486434d34ebbaa88618f3ce7655d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46593 } } peers { permanent_uuid: "511175ba711842d0938410818d4e01d5" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 33707 } }
I20260806 02:50:48.938900 22458 consensus_queue.cc:260] T 8ece3179d99b427aa83536200cc1872f P 511175ba711842d0938410818d4e01d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1664d39ba29483dbded606a5ed784af" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42977 } } peers { permanent_uuid: "4bc8486434d34ebbaa88618f3ce7655d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46593 } } peers { permanent_uuid: "511175ba711842d0938410818d4e01d5" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 33707 } }
I20260806 02:50:48.940908 22434 heartbeater.cc:503] Master 127.1.210.126:35957 was elected leader, sending a full tablet report...
I20260806 02:50:48.940909 22302 heartbeater.cc:503] Master 127.1.210.126:35957 was elected leader, sending a full tablet report...
I20260806 02:50:48.941109 22454 ts_tablet_manager.cc:1428] T 8ece3179d99b427aa83536200cc1872f P f1664d39ba29483dbded606a5ed784af: Time spent starting tablet: real 0.017s user 0.010s sys 0.004s
I20260806 02:50:48.941376 22458 ts_tablet_manager.cc:1428] T 8ece3179d99b427aa83536200cc1872f P 511175ba711842d0938410818d4e01d5: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260806 02:50:48.967777 22461 raft_consensus.cc:482] T 8ece3179d99b427aa83536200cc1872f P f1664d39ba29483dbded606a5ed784af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:50:48.968190 22461 raft_consensus.cc:504] T 8ece3179d99b427aa83536200cc1872f P f1664d39ba29483dbded606a5ed784af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1664d39ba29483dbded606a5ed784af" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42977 } } peers { permanent_uuid: "4bc8486434d34ebbaa88618f3ce7655d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46593 } } peers { permanent_uuid: "511175ba711842d0938410818d4e01d5" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 33707 } }
I20260806 02:50:48.970093 22461 leader_election.cc:288] T 8ece3179d99b427aa83536200cc1872f P f1664d39ba29483dbded606a5ed784af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bc8486434d34ebbaa88618f3ce7655d (127.1.210.65:46593), 511175ba711842d0938410818d4e01d5 (127.1.210.66:33707)
I20260806 02:50:48.978420 22450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:48.970870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:59505 (local address 127.1.210.65:46593)
0806 02:50:48.971104 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:48.971110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:48.971485 (+ 375us) server_negotiation.cc:407] Connection header received
0806 02:50:48.971588 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:48.971596 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:48.971705 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:48.971836 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:48.972533 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.973792 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.976293 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.977381 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.977525 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:48.977577 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:48.977980 (+ 403us) server_negotiation.cc:299] Negotiation successful
0806 02:50:48.978098 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 02:50:48.978466 22437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:48.970482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:46593 (local address 127.1.210.67:59505)
0806 02:50:48.971128 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:48.971152 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:48.971326 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:48.972039 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:48.972045 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:48.972074 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:48.972327 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:48.972341 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.974157 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.974163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:48.976101 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:48.976115 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.977552 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.977559 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:48.977751 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:48.977795 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 02:50:48.977907 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 02:50:48.978079 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":1}
I20260806 02:50:48.979440 22451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:48.970562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.210.67:52621 (local address 127.1.210.66:33707)
0806 02:50:48.970817 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:48.970825 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:48.972297 (+ 1472us) server_negotiation.cc:407] Connection header received
0806 02:50:48.972624 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:48.972630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:48.972739 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:48.972812 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:50:48.973765 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.974969 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.977053 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.978221 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.978404 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:48.978487 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:48.979069 (+ 582us) server_negotiation.cc:299] Negotiation successful
0806 02:50:48.979217 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 02:50:48.979537 22463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:48.970836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.66:33707 (local address 127.1.210.67:52621)
0806 02:50:48.972180 (+ 1344us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:48.972206 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:48.972397 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:48.973040 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:48.973045 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:48.973071 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:50:48.973528 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:48.973547 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.975155 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.975160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:48.976824 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:48.976840 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:48.978440 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:48.978456 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:48.978873 (+ 417us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:48.978911 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 02:50:48.979002 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 02:50:48.979147 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1197,"mutex_wait_us":22,"thread_start_us":488,"threads_started":1}
I20260806 02:50:48.979743 22121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ece3179d99b427aa83536200cc1872f" candidate_uuid: "f1664d39ba29483dbded606a5ed784af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc8486434d34ebbaa88618f3ce7655d" is_pre_election: true
I20260806 02:50:48.980400 22121 raft_consensus.cc:2454] T 8ece3179d99b427aa83536200cc1872f P 4bc8486434d34ebbaa88618f3ce7655d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1664d39ba29483dbded606a5ed784af in term 0.
I20260806 02:50:48.980509 22257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ece3179d99b427aa83536200cc1872f" candidate_uuid: "f1664d39ba29483dbded606a5ed784af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "511175ba711842d0938410818d4e01d5" is_pre_election: true
I20260806 02:50:48.981098 22257 raft_consensus.cc:2454] T 8ece3179d99b427aa83536200cc1872f P 511175ba711842d0938410818d4e01d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1664d39ba29483dbded606a5ed784af in term 0.
I20260806 02:50:48.981269 22321 leader_election.cc:302] T 8ece3179d99b427aa83536200cc1872f P f1664d39ba29483dbded606a5ed784af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc8486434d34ebbaa88618f3ce7655d, f1664d39ba29483dbded606a5ed784af; no voters:
I20260806 02:50:48.981758 22461 raft_consensus.cc:2781] T 8ece3179d99b427aa83536200cc1872f P f1664d39ba29483dbded606a5ed784af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:50:48.981932 22461 raft_consensus.cc:482] T 8ece3179d99b427aa83536200cc1872f P f1664d39ba29483dbded606a5ed784af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:50:48.982059 22461 raft_consensus.cc:3037] T 8ece3179d99b427aa83536200cc1872f P f1664d39ba29483dbded606a5ed784af [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:48.984494 22461 raft_consensus.cc:504] T 8ece3179d99b427aa83536200cc1872f P f1664d39ba29483dbded606a5ed784af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1664d39ba29483dbded606a5ed784af" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42977 } } peers { permanent_uuid: "4bc8486434d34ebbaa88618f3ce7655d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46593 } } peers { permanent_uuid: "511175ba711842d0938410818d4e01d5" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 33707 } }
I20260806 02:50:48.985114 22461 leader_election.cc:288] T 8ece3179d99b427aa83536200cc1872f P f1664d39ba29483dbded606a5ed784af [CANDIDATE]: Term 1 election: Requested vote from peers 4bc8486434d34ebbaa88618f3ce7655d (127.1.210.65:46593), 511175ba711842d0938410818d4e01d5 (127.1.210.66:33707)
I20260806 02:50:48.985730 22257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ece3179d99b427aa83536200cc1872f" candidate_uuid: "f1664d39ba29483dbded606a5ed784af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "511175ba711842d0938410818d4e01d5"
I20260806 02:50:48.985805 22121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ece3179d99b427aa83536200cc1872f" candidate_uuid: "f1664d39ba29483dbded606a5ed784af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc8486434d34ebbaa88618f3ce7655d"
I20260806 02:50:48.986027 22257 raft_consensus.cc:3037] T 8ece3179d99b427aa83536200cc1872f P 511175ba711842d0938410818d4e01d5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:48.986061 22121 raft_consensus.cc:3037] T 8ece3179d99b427aa83536200cc1872f P 4bc8486434d34ebbaa88618f3ce7655d [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:48.988487 22257 raft_consensus.cc:2454] T 8ece3179d99b427aa83536200cc1872f P 511175ba711842d0938410818d4e01d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1664d39ba29483dbded606a5ed784af in term 1.
I20260806 02:50:48.988663 22121 raft_consensus.cc:2454] T 8ece3179d99b427aa83536200cc1872f P 4bc8486434d34ebbaa88618f3ce7655d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1664d39ba29483dbded606a5ed784af in term 1.
I20260806 02:50:48.989151 22322 leader_election.cc:302] T 8ece3179d99b427aa83536200cc1872f P f1664d39ba29483dbded606a5ed784af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 511175ba711842d0938410818d4e01d5, f1664d39ba29483dbded606a5ed784af; no voters:
I20260806 02:50:48.989588 22461 raft_consensus.cc:2781] T 8ece3179d99b427aa83536200cc1872f P f1664d39ba29483dbded606a5ed784af [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:48.990226 22461 raft_consensus.cc:686] T 8ece3179d99b427aa83536200cc1872f P f1664d39ba29483dbded606a5ed784af [term 1 LEADER]: Becoming Leader. State: Replica: f1664d39ba29483dbded606a5ed784af, State: Running, Role: LEADER
I20260806 02:50:48.990826 22461 consensus_queue.cc:237] T 8ece3179d99b427aa83536200cc1872f P f1664d39ba29483dbded606a5ed784af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1664d39ba29483dbded606a5ed784af" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42977 } } peers { permanent_uuid: "4bc8486434d34ebbaa88618f3ce7655d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46593 } } peers { permanent_uuid: "511175ba711842d0938410818d4e01d5" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 33707 } }
I20260806 02:50:48.996182 21978 catalog_manager.cc:5495] T 8ece3179d99b427aa83536200cc1872f P f1664d39ba29483dbded606a5ed784af reported cstate change: term changed from 0 to 1, leader changed from <none> to f1664d39ba29483dbded606a5ed784af (127.1.210.67). New cstate: current_term: 1 leader_uuid: "f1664d39ba29483dbded606a5ed784af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1664d39ba29483dbded606a5ed784af" member_type: VOTER last_known_addr { host: "127.1.210.67" port: 42977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bc8486434d34ebbaa88618f3ce7655d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "511175ba711842d0938410818d4e01d5" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 33707 } health_report { overall_health: UNKNOWN } } }
W20260806 02:50:49.017760 22168 tablet.cc:2369] T 8ece3179d99b427aa83536200cc1872f P 4bc8486434d34ebbaa88618f3ce7655d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:50:49.018960 22303 tablet.cc:2369] T 8ece3179d99b427aa83536200cc1872f P 511175ba711842d0938410818d4e01d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:50:49.038892 22435 tablet.cc:2369] T 8ece3179d99b427aa83536200cc1872f P f1664d39ba29483dbded606a5ed784af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:50:49.043361 22449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:49.037233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39506 (local address 127.1.210.67:42977)
0806 02:50:49.037568 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:49.037575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:49.037597 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:50:49.037824 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:49.037829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:49.037919 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:49.038085 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:50:49.038484 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:49.039686 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:49.041078 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:49.041517 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:49.041655 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:49.042195 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:50:49.042792 (+ 597us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:50:49.042957 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:49.043010 (+ 53us) server_negotiation.cc:299] Negotiation successful
0806 02:50:49.043095 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260806 02:50:49.052102 22257 raft_consensus.cc:1261] T 8ece3179d99b427aa83536200cc1872f P 511175ba711842d0938410818d4e01d5 [term 1 FOLLOWER]: Refusing update from remote peer f1664d39ba29483dbded606a5ed784af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:50:49.052186 22121 raft_consensus.cc:1261] T 8ece3179d99b427aa83536200cc1872f P 4bc8486434d34ebbaa88618f3ce7655d [term 1 FOLLOWER]: Refusing update from remote peer f1664d39ba29483dbded606a5ed784af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:50:49.053159 22464 consensus_queue.cc:1035] T 8ece3179d99b427aa83536200cc1872f P f1664d39ba29483dbded606a5ed784af [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bc8486434d34ebbaa88618f3ce7655d" member_type: VOTER last_known_addr { host: "127.1.210.65" port: 46593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:50:49.053514 22461 consensus_queue.cc:1035] T 8ece3179d99b427aa83536200cc1872f P f1664d39ba29483dbded606a5ed784af [LEADER]: Connected to new peer: Peer: permanent_uuid: "511175ba711842d0938410818d4e01d5" member_type: VOTER last_known_addr { host: "127.1.210.66" port: 33707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:50:49.080355 21977 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:50:49.080675 21977 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:50:49.085233 21977 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 108a20824fa64ae3b989a1dfcb1cd04f: Sending DeleteTablet for 3 replicas of tablet 8ece3179d99b427aa83536200cc1872f
I20260806 02:50:49.086622 22368 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ece3179d99b427aa83536200cc1872f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:49 UTC) from {username='slave'} at 127.0.0.1:39496
I20260806 02:50:49.086733 22235 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ece3179d99b427aa83536200cc1872f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:49 UTC) from {username='slave'} at 127.0.0.1:58142
I20260806 02:50:49.086699 22100 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ece3179d99b427aa83536200cc1872f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:50:49 UTC) from {username='slave'} at 127.0.0.1:43620
I20260806 02:50:49.089229 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 22038
I20260806 02:50:49.098510 22480 tablet_replica.cc:330] T 8ece3179d99b427aa83536200cc1872f P 4bc8486434d34ebbaa88618f3ce7655d: stopping tablet replica
I20260806 02:50:49.099223 22480 raft_consensus.cc:2229] T 8ece3179d99b427aa83536200cc1872f P 4bc8486434d34ebbaa88618f3ce7655d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:50:49.099975 22480 raft_consensus.cc:2258] T 8ece3179d99b427aa83536200cc1872f P 4bc8486434d34ebbaa88618f3ce7655d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:49.102593 22479 tablet_replica.cc:330] T 8ece3179d99b427aa83536200cc1872f P 511175ba711842d0938410818d4e01d5: stopping tablet replica
I20260806 02:50:49.103332 22479 raft_consensus.cc:2229] T 8ece3179d99b427aa83536200cc1872f P 511175ba711842d0938410818d4e01d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:50:49.104034 22479 raft_consensus.cc:2258] T 8ece3179d99b427aa83536200cc1872f P 511175ba711842d0938410818d4e01d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:49.106652 22480 ts_tablet_manager.cc:1905] T 8ece3179d99b427aa83536200cc1872f P 4bc8486434d34ebbaa88618f3ce7655d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:49.109539 22479 ts_tablet_manager.cc:1905] T 8ece3179d99b427aa83536200cc1872f P 511175ba711842d0938410818d4e01d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:49.114409 22450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:49.091268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43622 (local address 127.1.210.65:46593)
0806 02:50:49.091538 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:49.091549 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:49.104811 (+ 13262us) server_negotiation.cc:407] Connection header received
0806 02:50:49.105387 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:49.105393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:49.105511 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:49.105595 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:49.106306 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:49.107394 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:49.109547 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:49.110138 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:49.110252 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:49.112189 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:49.112203 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:49.112227 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:49.113097 (+ 870us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:49.113171 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:49.113200 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:49.113317 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:49.113770 (+ 453us) server_negotiation.cc:299] Negotiation successful
0806 02:50:49.113968 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260806 02:50:49.116143 22162 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:50:49.117854 22479 ts_tablet_manager.cc:1918] T 8ece3179d99b427aa83536200cc1872f P 511175ba711842d0938410818d4e01d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:50:49.118073 22479 log.cc:1196] T 8ece3179d99b427aa83536200cc1872f P 511175ba711842d0938410818d4e01d5: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-1/wal/wals/8ece3179d99b427aa83536200cc1872f
I20260806 02:50:49.118856 22479 ts_tablet_manager.cc:1939] T 8ece3179d99b427aa83536200cc1872f P 511175ba711842d0938410818d4e01d5: Deleting consensus metadata
I20260806 02:50:49.121565 21964 catalog_manager.cc:4841] TS 511175ba711842d0938410818d4e01d5 (127.1.210.66:33707): tablet 8ece3179d99b427aa83536200cc1872f (table test-table [id=49c0f319f15e48819aedf8d14ee8a5fa]) successfully deleted
I20260806 02:50:49.122576 22478 tablet_replica.cc:330] T 8ece3179d99b427aa83536200cc1872f P f1664d39ba29483dbded606a5ed784af: stopping tablet replica
I20260806 02:50:49.123061 22478 raft_consensus.cc:2229] T 8ece3179d99b427aa83536200cc1872f P f1664d39ba29483dbded606a5ed784af [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:50:49.123682 22478 raft_consensus.cc:2258] T 8ece3179d99b427aa83536200cc1872f P f1664d39ba29483dbded606a5ed784af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:50:49.126729 22478 ts_tablet_manager.cc:1905] T 8ece3179d99b427aa83536200cc1872f P f1664d39ba29483dbded606a5ed784af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:50:49.137643 22478 ts_tablet_manager.cc:1918] T 8ece3179d99b427aa83536200cc1872f P f1664d39ba29483dbded606a5ed784af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:50:49.138037 22478 log.cc:1196] T 8ece3179d99b427aa83536200cc1872f P f1664d39ba29483dbded606a5ed784af: Deleting WAL directory at /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984304136196-1865-0/minicluster-data/ts-2/wal/wals/8ece3179d99b427aa83536200cc1872f
I20260806 02:50:49.138955 22478 ts_tablet_manager.cc:1939] T 8ece3179d99b427aa83536200cc1872f P f1664d39ba29483dbded606a5ed784af: Deleting consensus metadata
I20260806 02:50:49.141485 21964 catalog_manager.cc:4841] TS f1664d39ba29483dbded606a5ed784af (127.1.210.67:42977): tablet 8ece3179d99b427aa83536200cc1872f (table test-table [id=49c0f319f15e48819aedf8d14ee8a5fa]) successfully deleted
I20260806 02:50:49.231542 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 22038
W20260806 02:50:49.259321 21963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.210.65:46593 (error 108)
I20260806 02:50:49.260128 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 22171
I20260806 02:50:49.261792 22447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:49.261197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:46593 (local address 127.0.0.1:43638)
0806 02:50:49.261474 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:49.261598 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260806 02:50:49.262197 21963 catalog_manager.cc:4584] TS 4bc8486434d34ebbaa88618f3ce7655d (127.1.210.65:46593): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8ece3179d99b427aa83536200cc1872f: Network error: Client connection negotiation failed: client connection to 127.1.210.65:46593: connect: Connection refused (error 111)
I20260806 02:50:49.269245 22451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:49.261011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58150 (local address 127.1.210.66:33707)
0806 02:50:49.261258 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:49.261266 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:49.261537 (+ 271us) server_negotiation.cc:407] Connection header received
0806 02:50:49.261738 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:49.261744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:49.261842 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:49.261964 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:49.262662 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:49.263685 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:49.265323 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:49.266009 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:49.266150 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:49.267497 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:49.267508 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:49.267527 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:49.268300 (+ 773us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:49.268365 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:49.268391 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:49.268526 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:49.268708 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 02:50:49.268831 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 02:50:49.270036 22297 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:50:49.316803 22447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:49.316230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:46593 (local address 127.0.0.1:43652)
0806 02:50:49.316500 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:49.316617 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260806 02:50:49.318034 22447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:49.317597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:46593 (local address 127.0.0.1:43662)
0806 02:50:49.317760 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:49.317858 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260806 02:50:49.362463 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 22171
I20260806 02:50:49.379103 22447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:49.378421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:46593 (local address 127.0.0.1:43672)
0806 02:50:49.378704 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:49.378823 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260806 02:50:49.380477 22447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:49.379960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:46593 (local address 127.0.0.1:43678)
0806 02:50:49.380162 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:49.380266 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 02:50:49.383435 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 22305
I20260806 02:50:49.391021 22449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:49.384255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39518 (local address 127.1.210.67:42977)
0806 02:50:49.384439 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:49.384446 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:49.384466 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:50:49.384720 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:49.384727 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:49.384828 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:49.384952 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:49.385370 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:49.386202 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:49.387690 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:49.388148 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:49.388300 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:49.389483 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:49.389492 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:49.389508 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:49.390151 (+ 643us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:49.390215 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:49.390236 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:49.390373 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:49.390519 (+ 146us) server_negotiation.cc:299] Negotiation successful
0806 02:50:49.390631 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 02:50:49.391850 22428 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:50:49.461647 22447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:49.461097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:46593 (local address 127.0.0.1:43694)
0806 02:50:49.461370 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:49.461485 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260806 02:50:49.462981 22447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:49.462476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.65:46593 (local address 127.0.0.1:43700)
0806 02:50:49.462768 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:49.462854 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.65:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260806 02:50:49.517664 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 22305
I20260806 02:50:49.538828 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 21946
I20260806 02:50:49.540110 22009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:50:49.657698 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 21946
2026-08-06T02:50:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3125 ms)
[----------] 6 tests from EncryptionPolicyTest (23786 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:50:49 dist-test-slave-8tdl krb5kdc[22490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:50:49 dist-test-slave-8tdl krb5kdc[22490](info): set up 2 sockets
Aug 06 02:50:49 dist-test-slave-8tdl krb5kdc[22490](info): commencing operation
krb5kdc: starting...
W20260806 02:50:50.846442 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.001s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:50:51 dist-test-slave-8tdl krb5kdc[22490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:50:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:50:51.182832 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46027
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:51.549741 22503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:51.550082 22503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:51.550269 22503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:51.557780 22503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:51.557916 22503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:51.557966 22503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:51.558094 22503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:51.558163 22503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:51.568831 22503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46027
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:50:51.570338 22503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:51.571861 22503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:51.584812 22511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:51.584851 22509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:51.585084 22508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:51.585965 22503 server_base.cc:1034] running on GCE node
I20260806 02:50:51.586920 22503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:51.588436 22503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:51.589665 22503 hybrid_clock.cc:648] HybridClock initialized: now 1785984651589610 us; error 51 us; skew 500 ppm
I20260806 02:50:51.590068 22503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:50:51 dist-test-slave-8tdl krb5kdc[22490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:51.593465 22503 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:50:51.595238 22503 webserver.cc:460] Webserver started at http://127.0.0.1:36351/ using document root <none> and password file <none>
I20260806 02:50:51.595971 22503 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:51.596127 22503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:51.596453 22503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:51.599712 22503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "f2f6e36bf6c74de39f98e2170ceead7b"
format_stamp: "Formatted at 2026-08-06 02:50:51 on dist-test-slave-8tdl"
server_key: "0c1a797ebdd4478f3a9d48924c458935"
server_key_iv: "2457e04e914b1b799327271f124073a4"
server_key_version: "encryptionkey@0"
I20260806 02:50:51.600560 22503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "f2f6e36bf6c74de39f98e2170ceead7b"
format_stamp: "Formatted at 2026-08-06 02:50:51 on dist-test-slave-8tdl"
server_key: "0c1a797ebdd4478f3a9d48924c458935"
server_key_iv: "2457e04e914b1b799327271f124073a4"
server_key_version: "encryptionkey@0"
I20260806 02:50:51.606261 22503 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260806 02:50:51.610070 22518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:51.611685 22503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:50:51.611912 22503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "f2f6e36bf6c74de39f98e2170ceead7b"
format_stamp: "Formatted at 2026-08-06 02:50:51 on dist-test-slave-8tdl"
server_key: "0c1a797ebdd4478f3a9d48924c458935"
server_key_iv: "2457e04e914b1b799327271f124073a4"
server_key_version: "encryptionkey@0"
I20260806 02:50:51.612161 22503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:51.635469 22503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:51.639971 22503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:51.640316 22503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:51.661037 22503 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46027
I20260806 02:50:51.661072 22570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46027 every 8 connection(s)
I20260806 02:50:51.662663 22503 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:51.666040 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 22503
I20260806 02:50:51.666286 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:50:51.667200 1865 external_mini_cluster.cc:1428] Setting key 2630535497fe6da510b762b8666fa31f
I20260806 02:50:51.667326 22571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:51.683158 22571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2f6e36bf6c74de39f98e2170ceead7b: Bootstrap starting.
Aug 06 02:50:51 dist-test-slave-8tdl krb5kdc[22490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 02:50:51.687497 22571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2f6e36bf6c74de39f98e2170ceead7b: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:51.688985 22571 log.cc:824] T 00000000000000000000000000000000 P f2f6e36bf6c74de39f98e2170ceead7b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:51.692744 22571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2f6e36bf6c74de39f98e2170ceead7b: No bootstrap required, opened a new log
I20260806 02:50:51.694219 22574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:51.669334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41265 (local address 127.0.0.1:46027)
0806 02:50:51.670198 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:51.670211 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:51.670258 (+ 47us) server_negotiation.cc:407] Connection header received
0806 02:50:51.672375 (+ 2117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:51.672399 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:51.672858 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:51.673244 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:51.675257 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:51.676626 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:51.678493 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:51.679300 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:51.679446 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:51.684497 (+ 5051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:51.684509 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:51.684532 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:51.684590 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:51.688963 (+ 4373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:51.690196 (+ 1233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:51.690204 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:51.690210 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:51.690394 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:51.690976 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:51.690981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:51.690984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:51.691697 (+ 713us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:51.692041 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:51.692417 (+ 376us) server_negotiation.cc:299] Negotiation successful
0806 02:50:51.692819 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":395,"threads_started":1}
I20260806 02:50:51.698616 1865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:50:51.702020 22571 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2f6e36bf6c74de39f98e2170ceead7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f6e36bf6c74de39f98e2170ceead7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46027 } }
I20260806 02:50:51.702486 22571 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2f6e36bf6c74de39f98e2170ceead7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:51.702623 22571 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2f6e36bf6c74de39f98e2170ceead7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2f6e36bf6c74de39f98e2170ceead7b, State: Initialized, Role: FOLLOWER
I20260806 02:50:51.703274 22571 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2f6e36bf6c74de39f98e2170ceead7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f6e36bf6c74de39f98e2170ceead7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46027 } }
I20260806 02:50:51.703598 22571 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2f6e36bf6c74de39f98e2170ceead7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:50:51.703773 22571 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2f6e36bf6c74de39f98e2170ceead7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:50:51.703990 22571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2f6e36bf6c74de39f98e2170ceead7b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:51.706875 22571 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2f6e36bf6c74de39f98e2170ceead7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f6e36bf6c74de39f98e2170ceead7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46027 } }
I20260806 02:50:51.707502 22571 leader_election.cc:302] T 00000000000000000000000000000000 P f2f6e36bf6c74de39f98e2170ceead7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2f6e36bf6c74de39f98e2170ceead7b; no voters:
I20260806 02:50:51.708264 22571 leader_election.cc:288] T 00000000000000000000000000000000 P f2f6e36bf6c74de39f98e2170ceead7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:50:51.708367 22581 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2f6e36bf6c74de39f98e2170ceead7b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:51.709502 22581 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2f6e36bf6c74de39f98e2170ceead7b [term 1 LEADER]: Becoming Leader. State: Replica: f2f6e36bf6c74de39f98e2170ceead7b, State: Running, Role: LEADER
I20260806 02:50:51.710175 22581 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2f6e36bf6c74de39f98e2170ceead7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f6e36bf6c74de39f98e2170ceead7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46027 } }
I20260806 02:50:51.710856 22571 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2f6e36bf6c74de39f98e2170ceead7b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:51.713424 22583 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2f6e36bf6c74de39f98e2170ceead7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2f6e36bf6c74de39f98e2170ceead7b. Latest consensus state: current_term: 1 leader_uuid: "f2f6e36bf6c74de39f98e2170ceead7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f6e36bf6c74de39f98e2170ceead7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46027 } } }
I20260806 02:50:51.713944 22583 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2f6e36bf6c74de39f98e2170ceead7b [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:51.718981 22582 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2f6e36bf6c74de39f98e2170ceead7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2f6e36bf6c74de39f98e2170ceead7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2f6e36bf6c74de39f98e2170ceead7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46027 } } }
I20260806 02:50:51.719370 22582 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2f6e36bf6c74de39f98e2170ceead7b [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:51.722668 22574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:51.703859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43135 (local address 127.0.0.1:46027)
0806 02:50:51.704056 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:51.704060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:51.704607 (+ 547us) server_negotiation.cc:407] Connection header received
0806 02:50:51.704750 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:51.704755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:51.704845 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:51.704952 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:51.706371 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:51.707564 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:51.709286 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:51.710034 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:51.710186 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:51.713672 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:51.713682 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:51.713687 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:51.713730 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:51.717245 (+ 3515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:51.718353 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:51.718359 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:51.718362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:51.718540 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:51.719330 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:51.719335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:51.719338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:51.719921 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:51.720336 (+ 415us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:51.721998 (+ 1662us) server_negotiation.cc:299] Negotiation successful
0806 02:50:51.722253 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 02:50:51.731128 22586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:50:51.737059 22586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:51.747365 22586 catalog_manager.cc:1347] Generated new cluster ID: 399cedc2b93f448884fec74c898df411
I20260806 02:50:51.747502 22586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:51.762789 22586 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:50:51.764870 22586 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:51.778934 22586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2f6e36bf6c74de39f98e2170ceead7b: Generated new TSK 0
I20260806 02:50:51.779886 22586 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:50:51.793893 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 22503
I20260806 02:50:51.795842 22566 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:50:51.892228 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 22503
W20260806 02:50:51.913664 22487 connection.cc:535] client connection to 127.0.0.1:46027 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46027): Connection reset by peer (error 104)
2026-08-06T02:50:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2223 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:50:52 dist-test-slave-8tdl krb5kdc[22603](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:50:52 dist-test-slave-8tdl krb5kdc[22603](info): set up 2 sockets
Aug 06 02:50:52 dist-test-slave-8tdl krb5kdc[22603](info): commencing operation
krb5kdc: starting...
W20260806 02:50:53.074345 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:50:53 dist-test-slave-8tdl krb5kdc[22603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:50:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:50:53.404237 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46497
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:53.778239 22618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:53.778637 22618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:53.778824 22618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:53.786929 22618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:53.787101 22618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:53.787205 22618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:53.787312 22618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:53.787387 22618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:53.797837 22618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46497
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:50:53.799324 22618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:53.800694 22618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:53.813201 22623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:53.813234 22624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:53.813125 22626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:53.814263 22618 server_base.cc:1034] running on GCE node
I20260806 02:50:53.815229 22618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:53.816795 22618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:53.818013 22618 hybrid_clock.cc:648] HybridClock initialized: now 1785984653817974 us; error 38 us; skew 500 ppm
I20260806 02:50:53.818387 22618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:50:53 dist-test-slave-8tdl krb5kdc[22603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:53.825722 22618 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:50:53.827370 22618 webserver.cc:460] Webserver started at http://127.0.0.1:41621/ using document root <none> and password file <none>
I20260806 02:50:53.828119 22618 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:53.828227 22618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:53.828500 22618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:53.831828 22618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "8306bad4d3a74486abc0c83a8466e381"
format_stamp: "Formatted at 2026-08-06 02:50:53 on dist-test-slave-8tdl"
server_key: "0e217e76d414e760408888159773f4ec"
server_key_iv: "75f32eb574318ed3d2472a30d3d5dfba"
server_key_version: "encryptionkey@0"
I20260806 02:50:53.832656 22618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "8306bad4d3a74486abc0c83a8466e381"
format_stamp: "Formatted at 2026-08-06 02:50:53 on dist-test-slave-8tdl"
server_key: "0e217e76d414e760408888159773f4ec"
server_key_iv: "75f32eb574318ed3d2472a30d3d5dfba"
server_key_version: "encryptionkey@0"
I20260806 02:50:53.838496 22618 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 02:50:53.842371 22633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:53.843963 22618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:50:53.844194 22618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "8306bad4d3a74486abc0c83a8466e381"
format_stamp: "Formatted at 2026-08-06 02:50:53 on dist-test-slave-8tdl"
server_key: "0e217e76d414e760408888159773f4ec"
server_key_iv: "75f32eb574318ed3d2472a30d3d5dfba"
server_key_version: "encryptionkey@0"
I20260806 02:50:53.844435 22618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:53.885567 22618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:53.890399 22618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:53.890748 22618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:53.911495 22618 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46497
I20260806 02:50:53.911558 22684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46497 every 8 connection(s)
I20260806 02:50:53.913012 22618 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:53.917312 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 22618
I20260806 02:50:53.917419 22685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:53.917557 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:50:53.918169 1865 external_mini_cluster.cc:1428] Setting key 240b545cfe3ecd4a6aa2a23fbd59dec6
I20260806 02:50:53.932579 22685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8306bad4d3a74486abc0c83a8466e381: Bootstrap starting.
I20260806 02:50:53.932581 22688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:53.920225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49761 (local address 127.0.0.1:46497)
0806 02:50:53.921163 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:53.921176 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:53.921223 (+ 47us) server_negotiation.cc:407] Connection header received
0806 02:50:53.923223 (+ 2000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:53.923247 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:53.923683 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:53.924118 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:53.924739 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:53.926110 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:53.927573 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:53.928267 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:53.928484 (+ 217us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:53.929947 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:53.929954 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:53.929973 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:53.930783 (+ 810us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:53.930882 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:53.930909 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:53.931082 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:53.931380 (+ 298us) server_negotiation.cc:299] Negotiation successful
0806 02:50:53.931573 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":449,"threads_started":1}
I20260806 02:50:53.936829 22685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8306bad4d3a74486abc0c83a8466e381: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:53.937147 1865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:50:53.938280 22685 log.cc:824] T 00000000000000000000000000000000 P 8306bad4d3a74486abc0c83a8466e381: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:53.942241 22685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8306bad4d3a74486abc0c83a8466e381: No bootstrap required, opened a new log
I20260806 02:50:53.949541 22688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:53.943017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43771 (local address 127.0.0.1:46497)
0806 02:50:53.943216 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:53.943224 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:53.943670 (+ 446us) server_negotiation.cc:407] Connection header received
0806 02:50:53.943895 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:53.943900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:53.944004 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:53.944117 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:53.944608 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:53.945409 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:53.946789 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:53.947299 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:53.947474 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:53.948676 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:53.948681 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:53.948685 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:53.948709 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:53.948750 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:53.948765 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:53.948911 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:53.949108 (+ 197us) server_negotiation.cc:299] Negotiation successful
0806 02:50:53.949164 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 02:50:53.952121 22685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8306bad4d3a74486abc0c83a8466e381 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8306bad4d3a74486abc0c83a8466e381" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46497 } }
I20260806 02:50:53.952543 22685 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8306bad4d3a74486abc0c83a8466e381 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:53.952687 22685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8306bad4d3a74486abc0c83a8466e381 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8306bad4d3a74486abc0c83a8466e381, State: Initialized, Role: FOLLOWER
I20260806 02:50:53.953368 22685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8306bad4d3a74486abc0c83a8466e381 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8306bad4d3a74486abc0c83a8466e381" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46497 } }
I20260806 02:50:53.953660 22685 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8306bad4d3a74486abc0c83a8466e381 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:50:53.953835 22685 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8306bad4d3a74486abc0c83a8466e381 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:50:53.954020 22685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8306bad4d3a74486abc0c83a8466e381 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:53.956755 22685 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8306bad4d3a74486abc0c83a8466e381 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8306bad4d3a74486abc0c83a8466e381" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46497 } }
I20260806 02:50:53.957429 22685 leader_election.cc:302] T 00000000000000000000000000000000 P 8306bad4d3a74486abc0c83a8466e381 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8306bad4d3a74486abc0c83a8466e381; no voters:
I20260806 02:50:53.958093 22685 leader_election.cc:288] T 00000000000000000000000000000000 P 8306bad4d3a74486abc0c83a8466e381 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:50:53.958281 22695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8306bad4d3a74486abc0c83a8466e381 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:53.959090 22695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8306bad4d3a74486abc0c83a8466e381 [term 1 LEADER]: Becoming Leader. State: Replica: 8306bad4d3a74486abc0c83a8466e381, State: Running, Role: LEADER
I20260806 02:50:53.959728 22695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8306bad4d3a74486abc0c83a8466e381 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8306bad4d3a74486abc0c83a8466e381" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46497 } }
I20260806 02:50:53.960700 22685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8306bad4d3a74486abc0c83a8466e381 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:53.963083 22697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8306bad4d3a74486abc0c83a8466e381 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8306bad4d3a74486abc0c83a8466e381. Latest consensus state: current_term: 1 leader_uuid: "8306bad4d3a74486abc0c83a8466e381" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8306bad4d3a74486abc0c83a8466e381" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46497 } } }
I20260806 02:50:53.963589 22697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8306bad4d3a74486abc0c83a8466e381 [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:53.963315 22696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8306bad4d3a74486abc0c83a8466e381 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8306bad4d3a74486abc0c83a8466e381" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8306bad4d3a74486abc0c83a8466e381" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46497 } } }
I20260806 02:50:53.963745 22696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8306bad4d3a74486abc0c83a8466e381 [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:53.968541 22702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:50:53.975158 22702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:53.985145 22702 catalog_manager.cc:1347] Generated new cluster ID: 2779ea9c50a8433e8e8d617c4fc55c97
I20260806 02:50:53.985276 22702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:54.017516 22702 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:50:54.019757 22702 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:54.033865 22702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8306bad4d3a74486abc0c83a8466e381: Generated new TSK 0
I20260806 02:50:54.035149 22702 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:50:54.049970 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 22618
W20260806 02:50:54.051680 22680 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49761
W20260806 02:50:54.052610 1865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 22618: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:50:54.052839 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 22618
W20260806 02:50:54.074556 22601 connection.cc:535] client connection to 127.0.0.1:46497 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46497): Connection reset by peer (error 104)
2026-08-06T02:50:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2160 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:50:54 dist-test-slave-8tdl krb5kdc[22716](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:50:54 dist-test-slave-8tdl krb5kdc[22716](info): set up 2 sockets
Aug 06 02:50:54 dist-test-slave-8tdl krb5kdc[22716](info): commencing operation
krb5kdc: starting...
W20260806 02:50:55.214409 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.001s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:50:55 dist-test-slave-8tdl krb5kdc[22716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:50:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:50:55.535214 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34719
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:55.892330 22729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:55.892714 22729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:55.892910 22729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:55.900619 22729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:55.900748 22729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:55.900861 22729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:55.900986 22729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:55.901068 22729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:55.912196 22729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34719
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:50:55.913615 22729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:55.915100 22729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:55.927821 22735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:55.927912 22737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:55.927961 22734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:55.929109 22729 server_base.cc:1034] running on GCE node
I20260806 02:50:55.930042 22729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:55.931790 22729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:55.933131 22729 hybrid_clock.cc:648] HybridClock initialized: now 1785984655933033 us; error 35 us; skew 500 ppm
I20260806 02:50:55.933561 22729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:50:55 dist-test-slave-8tdl krb5kdc[22716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:55.937235 22729 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:50:55.939002 22729 webserver.cc:460] Webserver started at http://127.0.0.1:46225/ using document root <none> and password file <none>
I20260806 02:50:55.939788 22729 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:55.939941 22729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:55.940291 22729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:55.943535 22729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "0cf9d16316da406fa18ad1ccac9aaa74"
format_stamp: "Formatted at 2026-08-06 02:50:55 on dist-test-slave-8tdl"
server_key: "7bfce5247cc04086e89a632d3bdea706"
server_key_iv: "f1a84338d149f8b95acb70d761f8985e"
server_key_version: "encryptionkey@0"
I20260806 02:50:55.944348 22729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "0cf9d16316da406fa18ad1ccac9aaa74"
format_stamp: "Formatted at 2026-08-06 02:50:55 on dist-test-slave-8tdl"
server_key: "7bfce5247cc04086e89a632d3bdea706"
server_key_iv: "f1a84338d149f8b95acb70d761f8985e"
server_key_version: "encryptionkey@0"
I20260806 02:50:55.949904 22729 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260806 02:50:55.953869 22744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:55.955397 22729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:50:55.955614 22729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "0cf9d16316da406fa18ad1ccac9aaa74"
format_stamp: "Formatted at 2026-08-06 02:50:55 on dist-test-slave-8tdl"
server_key: "7bfce5247cc04086e89a632d3bdea706"
server_key_iv: "f1a84338d149f8b95acb70d761f8985e"
server_key_version: "encryptionkey@0"
I20260806 02:50:55.955827 22729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:55.962127 22729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:55.966741 22729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:55.967074 22729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:55.987303 22729 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34719
I20260806 02:50:55.987346 22797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34719 every 8 connection(s)
I20260806 02:50:55.988847 22729 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:55.993165 22798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:55.997331 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 22729
I20260806 02:50:55.997546 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:50:55.998191 1865 external_mini_cluster.cc:1428] Setting key 51d6cf0e56ea6aacc2b0490711f48d2c
I20260806 02:50:56.006613 22798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cf9d16316da406fa18ad1ccac9aaa74: Bootstrap starting.
I20260806 02:50:56.008893 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:56.000847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60913 (local address 127.0.0.1:34719)
0806 02:50:56.002289 (+ 1442us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:56.002302 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:56.002347 (+ 45us) server_negotiation.cc:407] Connection header received
0806 02:50:56.004787 (+ 2440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:56.004819 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:56.005254 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:56.005671 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:56.006957 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:56.006964 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:56.006981 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:56.006997 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:56.007068 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:56.007090 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:56.007278 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:56.007424 (+ 146us) server_negotiation.cc:299] Negotiation successful
0806 02:50:56.007666 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1187,"thread_start_us":744,"threads_started":1}
I20260806 02:50:56.010339 22798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cf9d16316da406fa18ad1ccac9aaa74: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:56.011619 22798 log.cc:824] T 00000000000000000000000000000000 P 0cf9d16316da406fa18ad1ccac9aaa74: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:56.013752 1865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:50:56.015821 22798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cf9d16316da406fa18ad1ccac9aaa74: No bootstrap required, opened a new log
I20260806 02:50:56.022436 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:56.019475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48379 (local address 127.0.0.1:34719)
0806 02:50:56.019704 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:56.019710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:56.020136 (+ 426us) server_negotiation.cc:407] Connection header received
0806 02:50:56.020307 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:56.020313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:56.020377 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:56.020501 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:56.021571 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:56.021586 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:56.021593 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:56.021619 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:56.021676 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:56.021703 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:56.021836 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:56.022035 (+ 199us) server_negotiation.cc:299] Negotiation successful
0806 02:50:56.022111 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 02:50:56.026463 22798 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cf9d16316da406fa18ad1ccac9aaa74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf9d16316da406fa18ad1ccac9aaa74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34719 } }
I20260806 02:50:56.026933 22798 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cf9d16316da406fa18ad1ccac9aaa74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:56.027019 22798 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cf9d16316da406fa18ad1ccac9aaa74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf9d16316da406fa18ad1ccac9aaa74, State: Initialized, Role: FOLLOWER
I20260806 02:50:56.027609 22798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cf9d16316da406fa18ad1ccac9aaa74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf9d16316da406fa18ad1ccac9aaa74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34719 } }
I20260806 02:50:56.027916 22798 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cf9d16316da406fa18ad1ccac9aaa74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:50:56.028086 22798 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cf9d16316da406fa18ad1ccac9aaa74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:50:56.028324 22798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cf9d16316da406fa18ad1ccac9aaa74 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:56.030841 22798 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cf9d16316da406fa18ad1ccac9aaa74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf9d16316da406fa18ad1ccac9aaa74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34719 } }
I20260806 02:50:56.031414 22798 leader_election.cc:302] T 00000000000000000000000000000000 P 0cf9d16316da406fa18ad1ccac9aaa74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cf9d16316da406fa18ad1ccac9aaa74; no voters:
I20260806 02:50:56.032033 22798 leader_election.cc:288] T 00000000000000000000000000000000 P 0cf9d16316da406fa18ad1ccac9aaa74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:50:56.032234 22808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cf9d16316da406fa18ad1ccac9aaa74 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:56.032948 22808 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cf9d16316da406fa18ad1ccac9aaa74 [term 1 LEADER]: Becoming Leader. State: Replica: 0cf9d16316da406fa18ad1ccac9aaa74, State: Running, Role: LEADER
I20260806 02:50:56.033530 22808 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cf9d16316da406fa18ad1ccac9aaa74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf9d16316da406fa18ad1ccac9aaa74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34719 } }
I20260806 02:50:56.034304 22798 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cf9d16316da406fa18ad1ccac9aaa74 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:56.036738 22809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf9d16316da406fa18ad1ccac9aaa74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0cf9d16316da406fa18ad1ccac9aaa74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf9d16316da406fa18ad1ccac9aaa74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34719 } } }
I20260806 02:50:56.037429 22809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf9d16316da406fa18ad1ccac9aaa74 [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:56.036725 22810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf9d16316da406fa18ad1ccac9aaa74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cf9d16316da406fa18ad1ccac9aaa74. Latest consensus state: current_term: 1 leader_uuid: "0cf9d16316da406fa18ad1ccac9aaa74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf9d16316da406fa18ad1ccac9aaa74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34719 } } }
I20260806 02:50:56.037884 22810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf9d16316da406fa18ad1ccac9aaa74 [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:56.040431 22814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:50:56.046329 22814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:56.057076 22814 catalog_manager.cc:1347] Generated new cluster ID: 2b260c56ef3e4a85835ad414de71c5c8
I20260806 02:50:56.057225 22814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:56.071525 22814 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:50:56.072010 22814 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:56.082149 22814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cf9d16316da406fa18ad1ccac9aaa74: Generated new TSK 0
I20260806 02:50:56.083315 22814 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:50:56.096591 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 22729
W20260806 02:50:56.098145 22793 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60913
W20260806 02:50:56.098955 1865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 22729: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:50:56.099107 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 22729
2026-08-06T02:50:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2045 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:50:56 dist-test-slave-8tdl krb5kdc[22830](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:50:56 dist-test-slave-8tdl krb5kdc[22830](info): set up 2 sockets
Aug 06 02:50:56 dist-test-slave-8tdl krb5kdc[22830](info): commencing operation
krb5kdc: starting...
W20260806 02:50:57.246487 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.008s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:50:57 dist-test-slave-8tdl krb5kdc[22830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:50:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:50:57.580094 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44199
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:57.943193 22843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:57.943501 22843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:57.943622 22843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:57.951213 22843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:57.951352 22843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:57.951400 22843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:57.951462 22843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:57.951506 22843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:57.961812 22843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:50:57.963236 22843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:57.964576 22843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:57.977283 22848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:57.977283 22849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:57.977401 22851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:57.979070 22843 server_base.cc:1034] running on GCE node
I20260806 02:50:57.980101 22843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:57.981789 22843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:57.983088 22843 hybrid_clock.cc:648] HybridClock initialized: now 1785984657983034 us; error 63 us; skew 500 ppm
I20260806 02:50:57.983559 22843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:50:57 dist-test-slave-8tdl krb5kdc[22830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:57.987213 22843 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:50:57.988953 22843 webserver.cc:460] Webserver started at http://127.0.0.1:39951/ using document root <none> and password file <none>
I20260806 02:50:57.989746 22843 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:57.989908 22843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:57.990257 22843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:57.993616 22843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "2dbc31c42722498cbad883e33a2c724f"
format_stamp: "Formatted at 2026-08-06 02:50:57 on dist-test-slave-8tdl"
server_key: "2af321bb4b93ccd849c73c128070252d"
server_key_iv: "36ac81dd447d20d35a0dbac8e28f358e"
server_key_version: "encryptionkey@0"
I20260806 02:50:57.994484 22843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "2dbc31c42722498cbad883e33a2c724f"
format_stamp: "Formatted at 2026-08-06 02:50:57 on dist-test-slave-8tdl"
server_key: "2af321bb4b93ccd849c73c128070252d"
server_key_iv: "36ac81dd447d20d35a0dbac8e28f358e"
server_key_version: "encryptionkey@0"
I20260806 02:50:58.000216 22843 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 02:50:58.004645 22858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:58.006410 22843 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260806 02:50:58.006649 22843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "2dbc31c42722498cbad883e33a2c724f"
format_stamp: "Formatted at 2026-08-06 02:50:57 on dist-test-slave-8tdl"
server_key: "2af321bb4b93ccd849c73c128070252d"
server_key_iv: "36ac81dd447d20d35a0dbac8e28f358e"
server_key_version: "encryptionkey@0"
I20260806 02:50:58.006979 22843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:58.023221 22843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:58.027921 22843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:58.028187 22843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:58.049141 22843 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44199
I20260806 02:50:58.049182 22911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44199 every 8 connection(s)
I20260806 02:50:58.050763 22843 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:58.055092 22912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:58.055663 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 22843
I20260806 02:50:58.055884 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:50:58.056715 1865 external_mini_cluster.cc:1428] Setting key 00d90b9161b9e6f263ed1638aa5a0f07
I20260806 02:50:58.070998 22912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dbc31c42722498cbad883e33a2c724f: Bootstrap starting.
Aug 06 02:50:58 dist-test-slave-8tdl krb5kdc[22830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 02:50:58.075114 22912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2dbc31c42722498cbad883e33a2c724f: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:58.076709 22912 log.cc:824] T 00000000000000000000000000000000 P 2dbc31c42722498cbad883e33a2c724f: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:58.080070 22912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dbc31c42722498cbad883e33a2c724f: No bootstrap required, opened a new log
I20260806 02:50:58.083213 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:58.058938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45969 (local address 127.0.0.1:44199)
0806 02:50:58.060036 (+ 1098us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:58.060053 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:58.060103 (+ 50us) server_negotiation.cc:407] Connection header received
0806 02:50:58.061986 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:58.062014 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:58.062506 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:58.062990 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:58.065259 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:58.066768 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:58.068657 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:58.069410 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:58.069548 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:58.074743 (+ 5195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:58.074763 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:58.074782 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:58.074831 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:58.078523 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:58.079399 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:58.079414 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:58.079421 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:58.079606 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:58.080043 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:58.080047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:58.080049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:58.080760 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:58.081140 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:58.081548 (+ 408us) server_negotiation.cc:299] Negotiation successful
0806 02:50:58.082031 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":456,"threads_started":1}
I20260806 02:50:58.087556 1865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:50:58.089298 22912 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dbc31c42722498cbad883e33a2c724f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dbc31c42722498cbad883e33a2c724f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44199 } }
I20260806 02:50:58.089695 22912 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2dbc31c42722498cbad883e33a2c724f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:58.089798 22912 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dbc31c42722498cbad883e33a2c724f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dbc31c42722498cbad883e33a2c724f, State: Initialized, Role: FOLLOWER
I20260806 02:50:58.090557 22912 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dbc31c42722498cbad883e33a2c724f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dbc31c42722498cbad883e33a2c724f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44199 } }
I20260806 02:50:58.090891 22912 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2dbc31c42722498cbad883e33a2c724f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:50:58.091061 22912 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2dbc31c42722498cbad883e33a2c724f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:50:58.091254 22912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dbc31c42722498cbad883e33a2c724f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:58.094810 22912 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2dbc31c42722498cbad883e33a2c724f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dbc31c42722498cbad883e33a2c724f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44199 } }
I20260806 02:50:58.095538 22912 leader_election.cc:302] T 00000000000000000000000000000000 P 2dbc31c42722498cbad883e33a2c724f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2dbc31c42722498cbad883e33a2c724f; no voters:
I20260806 02:50:58.096359 22912 leader_election.cc:288] T 00000000000000000000000000000000 P 2dbc31c42722498cbad883e33a2c724f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:50:58.096503 22922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2dbc31c42722498cbad883e33a2c724f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:58.097863 22922 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2dbc31c42722498cbad883e33a2c724f [term 1 LEADER]: Becoming Leader. State: Replica: 2dbc31c42722498cbad883e33a2c724f, State: Running, Role: LEADER
I20260806 02:50:58.098591 22922 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dbc31c42722498cbad883e33a2c724f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dbc31c42722498cbad883e33a2c724f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44199 } }
I20260806 02:50:58.099148 22912 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dbc31c42722498cbad883e33a2c724f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:58.104187 22924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dbc31c42722498cbad883e33a2c724f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dbc31c42722498cbad883e33a2c724f. Latest consensus state: current_term: 1 leader_uuid: "2dbc31c42722498cbad883e33a2c724f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dbc31c42722498cbad883e33a2c724f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44199 } } }
I20260806 02:50:58.104733 22924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dbc31c42722498cbad883e33a2c724f [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:58.105437 22923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dbc31c42722498cbad883e33a2c724f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2dbc31c42722498cbad883e33a2c724f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dbc31c42722498cbad883e33a2c724f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44199 } } }
I20260806 02:50:58.105908 22923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dbc31c42722498cbad883e33a2c724f [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:58.107272 22928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:50:58.112146 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:58.092684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54091 (local address 127.0.0.1:44199)
0806 02:50:58.092941 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:58.092949 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:58.093280 (+ 331us) server_negotiation.cc:407] Connection header received
0806 02:50:58.093456 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:58.093460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:58.093534 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:58.093658 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:58.095860 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:58.096834 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:58.098718 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:58.099341 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:58.099493 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:58.102290 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:58.102301 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:58.102309 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:58.102370 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:58.107339 (+ 4969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:58.108610 (+ 1271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:58.108616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:58.108619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:58.108792 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:58.109339 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:58.109348 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:58.109354 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:58.110144 (+ 790us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:58.110971 (+ 827us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:58.111186 (+ 215us) server_negotiation.cc:299] Negotiation successful
0806 02:50:58.111547 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260806 02:50:58.117020 22928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:58.129765 22928 catalog_manager.cc:1347] Generated new cluster ID: 54ad6212fddb49e9980ae5679b14af53
I20260806 02:50:58.129910 22928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:58.161492 22928 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:50:58.163476 22928 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:58.174464 22928 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2dbc31c42722498cbad883e33a2c724f: Generated new TSK 0
I20260806 02:50:58.175702 22928 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:50:58.194810 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 22843
I20260806 02:50:58.196130 22907 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:50:58.296077 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 22843
2026-08-06T02:50:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2197 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:50:58 dist-test-slave-8tdl krb5kdc[22945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:50:58 dist-test-slave-8tdl krb5kdc[22945](info): set up 2 sockets
Aug 06 02:50:58 dist-test-slave-8tdl krb5kdc[22945](info): commencing operation
krb5kdc: starting...
W20260806 02:50:59.487574 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.000s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:50:59 dist-test-slave-8tdl krb5kdc[22945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:50:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:50:59.826247 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37281
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:00.202121 22958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:00.202489 22958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:00.202692 22958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:00.210318 22958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:51:00.210507 22958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:00.210630 22958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:00.210773 22958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:51:00.210855 22958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:51:00.221383 22958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:51:00.222916 22958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:00.224327 22958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:00.237190 22963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:00.237282 22967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:00.237424 22964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:00.238855 22958 server_base.cc:1034] running on GCE node
I20260806 02:51:00.239879 22958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:00.241626 22958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:00.242887 22958 hybrid_clock.cc:648] HybridClock initialized: now 1785984660242841 us; error 53 us; skew 500 ppm
I20260806 02:51:00.243322 22958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:51:00 dist-test-slave-8tdl krb5kdc[22945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:00.247452 22958 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:51:00.249325 22958 webserver.cc:460] Webserver started at http://127.0.0.1:37179/ using document root <none> and password file <none>
I20260806 02:51:00.250149 22958 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:00.250321 22958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:00.250764 22958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:00.254225 22958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "ea53f93731854757a9c401989a63d57e"
format_stamp: "Formatted at 2026-08-06 02:51:00 on dist-test-slave-8tdl"
server_key: "066d4e6f15e06b90aeb1287e6b4fdaf7"
server_key_iv: "8679502267a0497877976d80989dc923"
server_key_version: "encryptionkey@0"
I20260806 02:51:00.255172 22958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "ea53f93731854757a9c401989a63d57e"
format_stamp: "Formatted at 2026-08-06 02:51:00 on dist-test-slave-8tdl"
server_key: "066d4e6f15e06b90aeb1287e6b4fdaf7"
server_key_iv: "8679502267a0497877976d80989dc923"
server_key_version: "encryptionkey@0"
I20260806 02:51:00.261253 22958 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260806 02:51:00.265486 22974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:00.267081 22958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:51:00.267319 22958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "ea53f93731854757a9c401989a63d57e"
format_stamp: "Formatted at 2026-08-06 02:51:00 on dist-test-slave-8tdl"
server_key: "066d4e6f15e06b90aeb1287e6b4fdaf7"
server_key_iv: "8679502267a0497877976d80989dc923"
server_key_version: "encryptionkey@0"
I20260806 02:51:00.267552 22958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:00.306340 22958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:00.311779 22958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:00.312143 22958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:00.333590 22958 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37281
I20260806 02:51:00.333634 23026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37281 every 8 connection(s)
I20260806 02:51:00.335289 22958 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:51:00.340164 23027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:00.340551 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 22958
I20260806 02:51:00.340754 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:51:00.341270 1865 external_mini_cluster.cc:1428] Setting key 2c4764453fca41ba849b02544165f0dd
I20260806 02:51:00.356184 23027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea53f93731854757a9c401989a63d57e: Bootstrap starting.
I20260806 02:51:00.358738 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.343386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45875 (local address 127.0.0.1:37281)
0806 02:51:00.344307 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:00.344323 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:00.344378 (+ 55us) server_negotiation.cc:407] Connection header received
0806 02:51:00.347477 (+ 3099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:00.347512 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:00.348118 (+ 606us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:00.348631 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:00.349195 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.351154 (+ 1959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.352807 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.353973 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.354154 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.355548 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:00.355559 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:00.355581 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:51:00.356363 (+ 782us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:00.356481 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:51:00.356531 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:00.356653 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:00.356960 (+ 307us) server_negotiation.cc:299] Negotiation successful
0806 02:51:00.357260 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":363,"threads_started":1}
I20260806 02:51:00.360479 23027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea53f93731854757a9c401989a63d57e: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:00.361756 23027 log.cc:824] T 00000000000000000000000000000000 P ea53f93731854757a9c401989a63d57e: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:00.363448 1865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:51:00.365386 23027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea53f93731854757a9c401989a63d57e: No bootstrap required, opened a new log
I20260806 02:51:00.376683 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:00.369169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58569 (local address 127.0.0.1:37281)
0806 02:51:00.369456 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:00.369465 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:00.369802 (+ 337us) server_negotiation.cc:407] Connection header received
0806 02:51:00.369979 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:00.369983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:00.370052 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:00.370191 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:00.370854 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.371914 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.373946 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:00.374482 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:00.374605 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:00.375904 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:00.375908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:00.375912 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:51:00.375932 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:00.375972 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:51:00.375989 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:00.376065 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:00.376231 (+ 166us) server_negotiation.cc:299] Negotiation successful
0806 02:51:00.376312 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 02:51:00.376863 23027 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea53f93731854757a9c401989a63d57e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea53f93731854757a9c401989a63d57e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37281 } }
I20260806 02:51:00.377373 23027 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea53f93731854757a9c401989a63d57e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:00.377504 23027 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea53f93731854757a9c401989a63d57e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea53f93731854757a9c401989a63d57e, State: Initialized, Role: FOLLOWER
I20260806 02:51:00.378144 23027 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea53f93731854757a9c401989a63d57e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea53f93731854757a9c401989a63d57e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37281 } }
I20260806 02:51:00.378422 23027 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea53f93731854757a9c401989a63d57e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:51:00.378579 23027 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea53f93731854757a9c401989a63d57e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:51:00.378823 23027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea53f93731854757a9c401989a63d57e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:00.381206 23027 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea53f93731854757a9c401989a63d57e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea53f93731854757a9c401989a63d57e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37281 } }
I20260806 02:51:00.381757 23027 leader_election.cc:302] T 00000000000000000000000000000000 P ea53f93731854757a9c401989a63d57e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea53f93731854757a9c401989a63d57e; no voters:
I20260806 02:51:00.382369 23027 leader_election.cc:288] T 00000000000000000000000000000000 P ea53f93731854757a9c401989a63d57e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:51:00.382534 23037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea53f93731854757a9c401989a63d57e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:51:00.383349 23037 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea53f93731854757a9c401989a63d57e [term 1 LEADER]: Becoming Leader. State: Replica: ea53f93731854757a9c401989a63d57e, State: Running, Role: LEADER
I20260806 02:51:00.384002 23037 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea53f93731854757a9c401989a63d57e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea53f93731854757a9c401989a63d57e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37281 } }
I20260806 02:51:00.384737 23027 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea53f93731854757a9c401989a63d57e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:51:00.387722 23038 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea53f93731854757a9c401989a63d57e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea53f93731854757a9c401989a63d57e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea53f93731854757a9c401989a63d57e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37281 } } }
I20260806 02:51:00.388001 23041 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea53f93731854757a9c401989a63d57e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea53f93731854757a9c401989a63d57e. Latest consensus state: current_term: 1 leader_uuid: "ea53f93731854757a9c401989a63d57e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea53f93731854757a9c401989a63d57e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37281 } } }
I20260806 02:51:00.388350 23041 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea53f93731854757a9c401989a63d57e [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:00.388224 23038 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea53f93731854757a9c401989a63d57e [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:00.390138 23047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:51:00.396056 23047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:51:00.405234 23047 catalog_manager.cc:1347] Generated new cluster ID: 70ae4edf67bc443989253765ba0d08df
I20260806 02:51:00.405375 23047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:51:00.417883 23047 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:51:00.419847 23047 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:51:00.428045 23047 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea53f93731854757a9c401989a63d57e: Generated new TSK 0
I20260806 02:51:00.429281 23047 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:51:00.447016 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 22958
W20260806 02:51:00.448276 23022 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45875
W20260806 02:51:00.449338 1865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 22958: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:51:00.449528 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 22958
2026-08-06T02:51:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2153 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:51:00 dist-test-slave-8tdl krb5kdc[23060](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:51:00 dist-test-slave-8tdl krb5kdc[23060](info): set up 2 sockets
Aug 06 02:51:00 dist-test-slave-8tdl krb5kdc[23060](info): commencing operation
krb5kdc: starting...
W20260806 02:51:01.614404 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.009s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:51:01 dist-test-slave-8tdl krb5kdc[23060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:51:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:51:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:51:01.946990 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43665
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:02.308895 23073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:02.309234 23073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:02.309407 23073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:02.317031 23073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:51:02.317164 23073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:02.317262 23073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:02.317368 23073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:51:02.317452 23073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:51:02.327836 23073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43665
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:51:02.329258 23073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:02.330715 23073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:02.343366 23081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:02.343362 23079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:02.343312 23078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:02.344281 23073 server_base.cc:1034] running on GCE node
I20260806 02:51:02.345093 23073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:02.346604 23073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:02.347867 23073 hybrid_clock.cc:648] HybridClock initialized: now 1785984662347792 us; error 74 us; skew 500 ppm
I20260806 02:51:02.348253 23073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:51:02 dist-test-slave-8tdl krb5kdc[23060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:02.351851 23073 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:51:02.353507 23073 webserver.cc:460] Webserver started at http://127.0.0.1:35695/ using document root <none> and password file <none>
I20260806 02:51:02.354267 23073 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:02.354401 23073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:02.354815 23073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:02.358012 23073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "b236717ef2c54ea48ef08d9bb92d49cc"
format_stamp: "Formatted at 2026-08-06 02:51:02 on dist-test-slave-8tdl"
server_key: "d6542430abf4f751db345c16abf08f1d"
server_key_iv: "43313c6f494e89911ae65983342bb272"
server_key_version: "encryptionkey@0"
I20260806 02:51:02.358918 23073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "b236717ef2c54ea48ef08d9bb92d49cc"
format_stamp: "Formatted at 2026-08-06 02:51:02 on dist-test-slave-8tdl"
server_key: "d6542430abf4f751db345c16abf08f1d"
server_key_iv: "43313c6f494e89911ae65983342bb272"
server_key_version: "encryptionkey@0"
I20260806 02:51:02.364499 23073 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 02:51:02.368460 23088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:02.370038 23073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:51:02.370235 23073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "b236717ef2c54ea48ef08d9bb92d49cc"
format_stamp: "Formatted at 2026-08-06 02:51:02 on dist-test-slave-8tdl"
server_key: "d6542430abf4f751db345c16abf08f1d"
server_key_iv: "43313c6f494e89911ae65983342bb272"
server_key_version: "encryptionkey@0"
I20260806 02:51:02.370461 23073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:02.376803 23073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:02.381553 23073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:02.381882 23073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:02.402114 23073 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43665
I20260806 02:51:02.402221 23141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43665 every 8 connection(s)
I20260806 02:51:02.403692 23073 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:51:02.407788 23142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:02.410065 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 23073
I20260806 02:51:02.410276 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:51:02.410893 1865 external_mini_cluster.cc:1428] Setting key fc7e0e1a81dedd7bf11e763c81daa537
I20260806 02:51:02.419960 23145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:02.412934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57605 (local address 127.0.0.1:43665)
0806 02:51:02.413926 (+ 992us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:02.413944 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:02.413992 (+ 48us) server_negotiation.cc:407] Connection header received
0806 02:51:02.416039 (+ 2047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:02.416066 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:02.416469 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:02.416844 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:02.418065 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:02.418072 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:02.418122 (+ 50us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:51:02.418144 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:02.418214 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:51:02.418247 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:02.418446 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:02.418569 (+ 123us) server_negotiation.cc:299] Negotiation successful
0806 02:51:02.418852 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":480,"threads_started":1}
I20260806 02:51:02.421496 23142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b236717ef2c54ea48ef08d9bb92d49cc: Bootstrap starting.
I20260806 02:51:02.424592 1865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:51:02.425112 23142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b236717ef2c54ea48ef08d9bb92d49cc: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:02.426190 23142 log.cc:824] T 00000000000000000000000000000000 P b236717ef2c54ea48ef08d9bb92d49cc: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:02.430153 23142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b236717ef2c54ea48ef08d9bb92d49cc: No bootstrap required, opened a new log
I20260806 02:51:02.432199 23145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:02.429452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33433 (local address 127.0.0.1:43665)
0806 02:51:02.429654 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:02.429662 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:02.430005 (+ 343us) server_negotiation.cc:407] Connection header received
0806 02:51:02.430217 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:02.430221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:02.430295 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:02.430389 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:02.431430 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:02.431436 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:02.431441 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:51:02.431459 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:02.431508 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:51:02.431523 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:02.431646 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:02.431751 (+ 105us) server_negotiation.cc:299] Negotiation successful
0806 02:51:02.431824 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:51:02.438959 23142 raft_consensus.cc:348] T 00000000000000000000000000000000 P b236717ef2c54ea48ef08d9bb92d49cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b236717ef2c54ea48ef08d9bb92d49cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43665 } }
I20260806 02:51:02.439431 23142 raft_consensus.cc:374] T 00000000000000000000000000000000 P b236717ef2c54ea48ef08d9bb92d49cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:02.439543 23142 raft_consensus.cc:729] T 00000000000000000000000000000000 P b236717ef2c54ea48ef08d9bb92d49cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b236717ef2c54ea48ef08d9bb92d49cc, State: Initialized, Role: FOLLOWER
I20260806 02:51:02.440220 23142 consensus_queue.cc:260] T 00000000000000000000000000000000 P b236717ef2c54ea48ef08d9bb92d49cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b236717ef2c54ea48ef08d9bb92d49cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43665 } }
I20260806 02:51:02.440481 23142 raft_consensus.cc:388] T 00000000000000000000000000000000 P b236717ef2c54ea48ef08d9bb92d49cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:51:02.440639 23142 raft_consensus.cc:482] T 00000000000000000000000000000000 P b236717ef2c54ea48ef08d9bb92d49cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:51:02.440812 23142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b236717ef2c54ea48ef08d9bb92d49cc [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:02.443779 23142 raft_consensus.cc:504] T 00000000000000000000000000000000 P b236717ef2c54ea48ef08d9bb92d49cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b236717ef2c54ea48ef08d9bb92d49cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43665 } }
I20260806 02:51:02.444324 23142 leader_election.cc:302] T 00000000000000000000000000000000 P b236717ef2c54ea48ef08d9bb92d49cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b236717ef2c54ea48ef08d9bb92d49cc; no voters:
I20260806 02:51:02.445005 23142 leader_election.cc:288] T 00000000000000000000000000000000 P b236717ef2c54ea48ef08d9bb92d49cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:51:02.445252 23152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b236717ef2c54ea48ef08d9bb92d49cc [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:51:02.446161 23152 raft_consensus.cc:686] T 00000000000000000000000000000000 P b236717ef2c54ea48ef08d9bb92d49cc [term 1 LEADER]: Becoming Leader. State: Replica: b236717ef2c54ea48ef08d9bb92d49cc, State: Running, Role: LEADER
I20260806 02:51:02.446869 23152 consensus_queue.cc:237] T 00000000000000000000000000000000 P b236717ef2c54ea48ef08d9bb92d49cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b236717ef2c54ea48ef08d9bb92d49cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43665 } }
I20260806 02:51:02.448021 23142 sys_catalog.cc:564] T 00000000000000000000000000000000 P b236717ef2c54ea48ef08d9bb92d49cc [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:51:02.450273 23154 sys_catalog.cc:455] T 00000000000000000000000000000000 P b236717ef2c54ea48ef08d9bb92d49cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b236717ef2c54ea48ef08d9bb92d49cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b236717ef2c54ea48ef08d9bb92d49cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43665 } } }
I20260806 02:51:02.450692 23153 sys_catalog.cc:455] T 00000000000000000000000000000000 P b236717ef2c54ea48ef08d9bb92d49cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b236717ef2c54ea48ef08d9bb92d49cc. Latest consensus state: current_term: 1 leader_uuid: "b236717ef2c54ea48ef08d9bb92d49cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b236717ef2c54ea48ef08d9bb92d49cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43665 } } }
I20260806 02:51:02.451057 23154 sys_catalog.cc:458] T 00000000000000000000000000000000 P b236717ef2c54ea48ef08d9bb92d49cc [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:02.451076 23153 sys_catalog.cc:458] T 00000000000000000000000000000000 P b236717ef2c54ea48ef08d9bb92d49cc [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:02.452940 23161 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:51:02.458647 23161 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:51:02.468812 23161 catalog_manager.cc:1347] Generated new cluster ID: c9373463a8e94662b318fb3fd26601d1
I20260806 02:51:02.468954 23161 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:51:02.489567 23161 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:51:02.490132 23161 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:51:02.506589 23161 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b236717ef2c54ea48ef08d9bb92d49cc: Generated new TSK 0
I20260806 02:51:02.507651 23161 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:51:02.520335 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 23073
W20260806 02:51:02.521520 23137 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57605
W20260806 02:51:02.522310 1865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 23073: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:51:02.522501 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 23073
2026-08-06T02:51:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2072 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:51:02 dist-test-slave-8tdl krb5kdc[23173](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:51:02 dist-test-slave-8tdl krb5kdc[23173](info): set up 2 sockets
Aug 06 02:51:02 dist-test-slave-8tdl krb5kdc[23173](info): commencing operation
krb5kdc: starting...
W20260806 02:51:03.682651 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.001s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:51:03 dist-test-slave-8tdl krb5kdc[23173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:51:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:51:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:51:04.026757 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45783
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:04.385355 23186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:04.385860 23186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:04.386059 23186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:04.393640 23186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:51:04.393759 23186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:04.393806 23186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:04.393862 23186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:51:04.393908 23186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:51:04.404259 23186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:51:04.405628 23186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:04.407127 23186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:04.420320 23192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:04.420472 23191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:04.420378 23194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:04.421986 23186 server_base.cc:1034] running on GCE node
I20260806 02:51:04.423034 23186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:04.424595 23186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:04.425851 23186 hybrid_clock.cc:648] HybridClock initialized: now 1785984664425805 us; error 38 us; skew 500 ppm
I20260806 02:51:04.426240 23186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:51:04 dist-test-slave-8tdl krb5kdc[23173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:04.433862 23186 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:51:04.435858 23186 webserver.cc:460] Webserver started at http://127.0.0.1:36217/ using document root <none> and password file <none>
I20260806 02:51:04.436659 23186 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:04.436825 23186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:04.437175 23186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:04.440470 23186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "f247ab624527482a8231b9d28f736996"
format_stamp: "Formatted at 2026-08-06 02:51:04 on dist-test-slave-8tdl"
server_key: "fb5a25c5bd7acd1a6fe0edc7b96cbc35"
server_key_iv: "1e49621d6f900938bcbaa179ab7c547c"
server_key_version: "encryptionkey@0"
I20260806 02:51:04.441308 23186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "f247ab624527482a8231b9d28f736996"
format_stamp: "Formatted at 2026-08-06 02:51:04 on dist-test-slave-8tdl"
server_key: "fb5a25c5bd7acd1a6fe0edc7b96cbc35"
server_key_iv: "1e49621d6f900938bcbaa179ab7c547c"
server_key_version: "encryptionkey@0"
I20260806 02:51:04.447249 23186 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 02:51:04.451097 23201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:04.452575 23186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:51:04.452772 23186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "f247ab624527482a8231b9d28f736996"
format_stamp: "Formatted at 2026-08-06 02:51:04 on dist-test-slave-8tdl"
server_key: "fb5a25c5bd7acd1a6fe0edc7b96cbc35"
server_key_iv: "1e49621d6f900938bcbaa179ab7c547c"
server_key_version: "encryptionkey@0"
I20260806 02:51:04.452989 23186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:04.468668 23186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:04.473600 23186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:04.473946 23186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:04.496160 23186 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45783
I20260806 02:51:04.496196 23255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45783 every 8 connection(s)
I20260806 02:51:04.497721 23186 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:51:04.501703 23256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:04.503244 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 23186
I20260806 02:51:04.503443 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:51:04.504001 1865 external_mini_cluster.cc:1428] Setting key d1700fef9750e73045cac7ed9346961f
I20260806 02:51:04.515295 23256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f247ab624527482a8231b9d28f736996: Bootstrap starting.
Aug 06 02:51:04 dist-test-slave-8tdl krb5kdc[23173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 02:51:04.518910 23256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f247ab624527482a8231b9d28f736996: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:04.520018 23256 log.cc:824] T 00000000000000000000000000000000 P f247ab624527482a8231b9d28f736996: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:04.523666 23256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f247ab624527482a8231b9d28f736996: No bootstrap required, opened a new log
I20260806 02:51:04.528694 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:04.506101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:58197 (local address 127.0.0.1:45783)
0806 02:51:04.506973 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:04.506986 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:04.507039 (+ 53us) server_negotiation.cc:407] Connection header received
0806 02:51:04.508732 (+ 1693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:04.508760 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:04.509210 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:04.509624 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:04.511278 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:04.512620 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:04.513987 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:04.514762 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:04.514976 (+ 214us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:04.519657 (+ 4681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:04.519666 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:04.519682 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:04.519725 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:04.523747 (+ 4022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:04.524892 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:04.524899 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:04.524906 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:04.525100 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:04.525643 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:04.525646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:04.525651 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:04.526237 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:04.526518 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:04.527115 (+ 597us) server_negotiation.cc:299] Negotiation successful
0806 02:51:04.527524 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":376,"threads_started":1}
I20260806 02:51:04.533031 1865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:51:04.532814 23256 raft_consensus.cc:348] T 00000000000000000000000000000000 P f247ab624527482a8231b9d28f736996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f247ab624527482a8231b9d28f736996" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45783 } }
I20260806 02:51:04.533293 23256 raft_consensus.cc:374] T 00000000000000000000000000000000 P f247ab624527482a8231b9d28f736996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:04.533394 23256 raft_consensus.cc:729] T 00000000000000000000000000000000 P f247ab624527482a8231b9d28f736996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f247ab624527482a8231b9d28f736996, State: Initialized, Role: FOLLOWER
I20260806 02:51:04.534032 23256 consensus_queue.cc:260] T 00000000000000000000000000000000 P f247ab624527482a8231b9d28f736996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f247ab624527482a8231b9d28f736996" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45783 } }
I20260806 02:51:04.534336 23256 raft_consensus.cc:388] T 00000000000000000000000000000000 P f247ab624527482a8231b9d28f736996 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:51:04.534471 23256 raft_consensus.cc:482] T 00000000000000000000000000000000 P f247ab624527482a8231b9d28f736996 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:51:04.534636 23256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f247ab624527482a8231b9d28f736996 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:04.538157 23256 raft_consensus.cc:504] T 00000000000000000000000000000000 P f247ab624527482a8231b9d28f736996 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f247ab624527482a8231b9d28f736996" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45783 } }
I20260806 02:51:04.538894 23256 leader_election.cc:302] T 00000000000000000000000000000000 P f247ab624527482a8231b9d28f736996 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f247ab624527482a8231b9d28f736996; no voters:
I20260806 02:51:04.539638 23256 leader_election.cc:288] T 00000000000000000000000000000000 P f247ab624527482a8231b9d28f736996 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:51:04.539774 23267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f247ab624527482a8231b9d28f736996 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:51:04.540967 23267 raft_consensus.cc:686] T 00000000000000000000000000000000 P f247ab624527482a8231b9d28f736996 [term 1 LEADER]: Becoming Leader. State: Replica: f247ab624527482a8231b9d28f736996, State: Running, Role: LEADER
I20260806 02:51:04.541721 23267 consensus_queue.cc:237] T 00000000000000000000000000000000 P f247ab624527482a8231b9d28f736996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f247ab624527482a8231b9d28f736996" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45783 } }
I20260806 02:51:04.541939 23256 sys_catalog.cc:564] T 00000000000000000000000000000000 P f247ab624527482a8231b9d28f736996 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:51:04.545064 23269 sys_catalog.cc:455] T 00000000000000000000000000000000 P f247ab624527482a8231b9d28f736996 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f247ab624527482a8231b9d28f736996. Latest consensus state: current_term: 1 leader_uuid: "f247ab624527482a8231b9d28f736996" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f247ab624527482a8231b9d28f736996" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45783 } } }
I20260806 02:51:04.545529 23269 sys_catalog.cc:458] T 00000000000000000000000000000000 P f247ab624527482a8231b9d28f736996 [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:04.547993 23268 sys_catalog.cc:455] T 00000000000000000000000000000000 P f247ab624527482a8231b9d28f736996 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f247ab624527482a8231b9d28f736996" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f247ab624527482a8231b9d28f736996" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45783 } } }
I20260806 02:51:04.548388 23268 sys_catalog.cc:458] T 00000000000000000000000000000000 P f247ab624527482a8231b9d28f736996 [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:04.549675 23273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:51:04.557142 23273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:51:04.562024 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:04.538624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:39473 (local address 127.0.0.1:45783)
0806 02:51:04.538869 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:04.538883 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:04.539407 (+ 524us) server_negotiation.cc:407] Connection header received
0806 02:51:04.539632 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:04.539644 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:04.539734 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:04.539852 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:04.542145 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:04.543424 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:04.545338 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:04.546063 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:04.546210 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:04.550625 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:04.550635 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:04.550641 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:04.550710 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:04.556128 (+ 5418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:04.557295 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:04.557304 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:04.557310 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:04.557495 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:04.558497 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:04.558506 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:04.558511 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:04.559162 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:04.560850 (+ 1688us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:04.561137 (+ 287us) server_negotiation.cc:299] Negotiation successful
0806 02:51:04.561424 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 02:51:04.568755 23273 catalog_manager.cc:1347] Generated new cluster ID: 3cd7b37d8acf4e32ba18b5f6ed4e90ca
I20260806 02:51:04.568897 23273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:51:04.581790 23273 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:51:04.583516 23273 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:51:04.593264 23273 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f247ab624527482a8231b9d28f736996: Generated new TSK 0
I20260806 02:51:04.594213 23273 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:51:04.600948 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 23186
I20260806 02:51:04.602139 23250 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:51:04.695564 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 23186
W20260806 02:51:04.716917 23171 connection.cc:535] client connection to 127.0.0.1:45783 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45783): Connection reset by peer (error 104)
2026-08-06T02:51:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2173 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:51:04 dist-test-slave-8tdl krb5kdc[23289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:51:04 dist-test-slave-8tdl krb5kdc[23289](info): set up 2 sockets
Aug 06 02:51:04 dist-test-slave-8tdl krb5kdc[23289](info): commencing operation
krb5kdc: starting...
W20260806 02:51:05.842331 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.002s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:51:06 dist-test-slave-8tdl krb5kdc[23289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:51:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:51:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:51:06.188360 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39127
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:06.549606 23302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:06.549945 23302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:06.550130 23302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:06.557911 23302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:51:06.558077 23302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:06.558128 23302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:06.558235 23302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:51:06.558303 23302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:51:06.569036 23302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:51:06.570462 23302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:06.571872 23302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:06.584196 23307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:06.584386 23308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:06.584347 23310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:06.585568 23302 server_base.cc:1034] running on GCE node
I20260806 02:51:06.586505 23302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:06.588177 23302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:06.589438 23302 hybrid_clock.cc:648] HybridClock initialized: now 1785984666589380 us; error 49 us; skew 500 ppm
I20260806 02:51:06.589819 23302 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:51:06 dist-test-slave-8tdl krb5kdc[23289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:06.593626 23302 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:51:06.595825 23302 webserver.cc:460] Webserver started at http://127.0.0.1:46659/ using document root <none> and password file <none>
I20260806 02:51:06.596586 23302 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:06.596747 23302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:06.597218 23302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:06.600711 23302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "2bbe0401d2fb42a9a075d315c5e905ab"
format_stamp: "Formatted at 2026-08-06 02:51:06 on dist-test-slave-8tdl"
server_key: "7ec0a469c4a79c53284e76b51d9e2962"
server_key_iv: "a5ed5ae8669d768b0e297cbaa37c4ced"
server_key_version: "encryptionkey@0"
I20260806 02:51:06.601583 23302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "2bbe0401d2fb42a9a075d315c5e905ab"
format_stamp: "Formatted at 2026-08-06 02:51:06 on dist-test-slave-8tdl"
server_key: "7ec0a469c4a79c53284e76b51d9e2962"
server_key_iv: "a5ed5ae8669d768b0e297cbaa37c4ced"
server_key_version: "encryptionkey@0"
I20260806 02:51:06.607348 23302 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 02:51:06.611181 23317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:06.612617 23302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:51:06.612816 23302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "2bbe0401d2fb42a9a075d315c5e905ab"
format_stamp: "Formatted at 2026-08-06 02:51:06 on dist-test-slave-8tdl"
server_key: "7ec0a469c4a79c53284e76b51d9e2962"
server_key_iv: "a5ed5ae8669d768b0e297cbaa37c4ced"
server_key_version: "encryptionkey@0"
I20260806 02:51:06.613056 23302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:06.627967 23302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:06.632834 23302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:06.633195 23302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:06.653364 23302 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39127
I20260806 02:51:06.653410 23369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39127 every 8 connection(s)
I20260806 02:51:06.654847 23302 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:51:06.659035 23370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:06.663329 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 23302
I20260806 02:51:06.663543 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:51:06.664008 1865 external_mini_cluster.cc:1428] Setting key 54ea8e43ee8db67902645c9f37b40348
I20260806 02:51:06.670970 23370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bbe0401d2fb42a9a075d315c5e905ab: Bootstrap starting.
I20260806 02:51:06.674929 23370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bbe0401d2fb42a9a075d315c5e905ab: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:06.676314 23370 log.cc:824] T 00000000000000000000000000000000 P 2bbe0401d2fb42a9a075d315c5e905ab: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:06.679926 23370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bbe0401d2fb42a9a075d315c5e905ab: No bootstrap required, opened a new log
I20260806 02:51:06.680086 23373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:06.666001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:35381 (local address 127.0.0.1:39127)
0806 02:51:06.666897 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:06.666909 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:06.666952 (+ 43us) server_negotiation.cc:407] Connection header received
0806 02:51:06.669429 (+ 2477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:06.669457 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:06.669932 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:06.670582 (+ 650us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:06.671269 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:06.672639 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:06.674775 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:06.675653 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:06.675801 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:06.677326 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:06.677337 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:06.677361 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:51:06.678176 (+ 815us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:06.678257 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:51:06.678290 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:06.678418 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:06.678831 (+ 413us) server_negotiation.cc:299] Negotiation successful
0806 02:51:06.679054 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":418,"threads_started":1}
I20260806 02:51:06.684398 1865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:51:06.688755 23370 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bbe0401d2fb42a9a075d315c5e905ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bbe0401d2fb42a9a075d315c5e905ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39127 } }
I20260806 02:51:06.689291 23370 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bbe0401d2fb42a9a075d315c5e905ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:06.689417 23370 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bbe0401d2fb42a9a075d315c5e905ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bbe0401d2fb42a9a075d315c5e905ab, State: Initialized, Role: FOLLOWER
I20260806 02:51:06.690166 23370 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bbe0401d2fb42a9a075d315c5e905ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bbe0401d2fb42a9a075d315c5e905ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39127 } }
I20260806 02:51:06.690496 23370 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bbe0401d2fb42a9a075d315c5e905ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:51:06.690680 23370 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bbe0401d2fb42a9a075d315c5e905ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:51:06.690888 23370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bbe0401d2fb42a9a075d315c5e905ab [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:06.693801 23370 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bbe0401d2fb42a9a075d315c5e905ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bbe0401d2fb42a9a075d315c5e905ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39127 } }
I20260806 02:51:06.694435 23370 leader_election.cc:302] T 00000000000000000000000000000000 P 2bbe0401d2fb42a9a075d315c5e905ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bbe0401d2fb42a9a075d315c5e905ab; no voters:
I20260806 02:51:06.695340 23370 leader_election.cc:288] T 00000000000000000000000000000000 P 2bbe0401d2fb42a9a075d315c5e905ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:51:06.695497 23380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bbe0401d2fb42a9a075d315c5e905ab [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:51:06.696897 23380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bbe0401d2fb42a9a075d315c5e905ab [term 1 LEADER]: Becoming Leader. State: Replica: 2bbe0401d2fb42a9a075d315c5e905ab, State: Running, Role: LEADER
I20260806 02:51:06.697278 23373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:06.689010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:56559 (local address 127.0.0.1:39127)
0806 02:51:06.689248 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:06.689258 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:06.689876 (+ 618us) server_negotiation.cc:407] Connection header received
0806 02:51:06.690084 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:06.690090 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:06.690184 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:06.690300 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:06.691036 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:06.692246 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:06.693569 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:06.694278 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:06.694409 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:06.695850 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:06.695860 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:06.695872 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:51:06.695907 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:06.696005 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:51:06.696032 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:06.696753 (+ 721us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:06.696840 (+ 87us) server_negotiation.cc:299] Negotiation successful
0806 02:51:06.696905 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 02:51:06.697563 23380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bbe0401d2fb42a9a075d315c5e905ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bbe0401d2fb42a9a075d315c5e905ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39127 } }
I20260806 02:51:06.697974 23370 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bbe0401d2fb42a9a075d315c5e905ab [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:51:06.701084 23382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bbe0401d2fb42a9a075d315c5e905ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bbe0401d2fb42a9a075d315c5e905ab. Latest consensus state: current_term: 1 leader_uuid: "2bbe0401d2fb42a9a075d315c5e905ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bbe0401d2fb42a9a075d315c5e905ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39127 } } }
I20260806 02:51:06.701627 23382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bbe0401d2fb42a9a075d315c5e905ab [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:06.701188 23381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bbe0401d2fb42a9a075d315c5e905ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bbe0401d2fb42a9a075d315c5e905ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bbe0401d2fb42a9a075d315c5e905ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39127 } } }
I20260806 02:51:06.701998 23381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bbe0401d2fb42a9a075d315c5e905ab [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:06.704126 23389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:51:06.709080 23389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:51:06.720440 23389 catalog_manager.cc:1347] Generated new cluster ID: be8a54b370514992a164658fc790cdca
I20260806 02:51:06.720641 23389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:51:06.737604 23389 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:51:06.739610 23389 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:51:06.751303 23389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bbe0401d2fb42a9a075d315c5e905ab: Generated new TSK 0
I20260806 02:51:06.752534 23389 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:51:06.761929 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 23302
W20260806 02:51:06.763829 23365 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.51:35381
W20260806 02:51:06.764690 1865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 23302: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:51:06.764853 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 23302
W20260806 02:51:06.785633 23287 connection.cc:535] client connection to 127.0.0.1:39127 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39127): Connection reset by peer (error 104)
2026-08-06T02:51:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2068 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:51:06 dist-test-slave-8tdl krb5kdc[23403](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:51:06 dist-test-slave-8tdl krb5kdc[23403](info): set up 2 sockets
Aug 06 02:51:06 dist-test-slave-8tdl krb5kdc[23403](info): commencing operation
krb5kdc: starting...
W20260806 02:51:07.950498 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.000s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:51:08 dist-test-slave-8tdl krb5kdc[23403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:51:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:51:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:51:08.288569 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41275
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:08.644228 23416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:08.644609 23416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:08.644793 23416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:08.652287 23416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:51:08.652460 23416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:08.652573 23416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:08.652684 23416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:51:08.652786 23416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:51:08.663138 23416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:51:08.664551 23416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:08.666114 23416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:08.679054 23421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:08.679059 23422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:08.679273 23424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:08.679648 23416 server_base.cc:1034] running on GCE node
I20260806 02:51:08.680670 23416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:08.682217 23416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:08.683452 23416 hybrid_clock.cc:648] HybridClock initialized: now 1785984668683398 us; error 48 us; skew 500 ppm
I20260806 02:51:08.683851 23416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:51:08 dist-test-slave-8tdl krb5kdc[23403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:08.687196 23416 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:51:08.688783 23416 webserver.cc:460] Webserver started at http://127.0.0.1:40803/ using document root <none> and password file <none>
I20260806 02:51:08.689476 23416 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:08.689589 23416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:08.689850 23416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:08.693037 23416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "713f4e35058648d28323ee60f8a44908"
format_stamp: "Formatted at 2026-08-06 02:51:08 on dist-test-slave-8tdl"
server_key: "1e50c896f77efe2392b1699043b43d65"
server_key_iv: "18d31732e0cff4e1c5275cca69d76f21"
server_key_version: "encryptionkey@0"
I20260806 02:51:08.693848 23416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "713f4e35058648d28323ee60f8a44908"
format_stamp: "Formatted at 2026-08-06 02:51:08 on dist-test-slave-8tdl"
server_key: "1e50c896f77efe2392b1699043b43d65"
server_key_iv: "18d31732e0cff4e1c5275cca69d76f21"
server_key_version: "encryptionkey@0"
I20260806 02:51:08.699472 23416 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260806 02:51:08.703225 23431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:08.704653 23416 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 02:51:08.704838 23416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "713f4e35058648d28323ee60f8a44908"
format_stamp: "Formatted at 2026-08-06 02:51:08 on dist-test-slave-8tdl"
server_key: "1e50c896f77efe2392b1699043b43d65"
server_key_iv: "18d31732e0cff4e1c5275cca69d76f21"
server_key_version: "encryptionkey@0"
I20260806 02:51:08.705068 23416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:08.711138 23416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:08.715737 23416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:08.716068 23416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:08.736359 23416 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41275
I20260806 02:51:08.736425 23482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41275 every 8 connection(s)
I20260806 02:51:08.737879 23416 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:51:08.741410 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 23416
I20260806 02:51:08.741617 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:51:08.742172 1865 external_mini_cluster.cc:1428] Setting key 347ae2bcdd54d409b89b43ba699e174f
I20260806 02:51:08.742424 23483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:08.751732 23486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:08.744269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:54649 (local address 127.0.0.1:41275)
0806 02:51:08.745147 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:08.745160 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:08.745203 (+ 43us) server_negotiation.cc:407] Connection header received
0806 02:51:08.747927 (+ 2724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:08.747953 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:08.748339 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:08.748843 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:08.749976 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:08.749986 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:08.750000 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:51:08.750017 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:08.750080 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:51:08.750106 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:08.750356 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:08.750519 (+ 163us) server_negotiation.cc:299] Negotiation successful
0806 02:51:08.750766 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":390,"threads_started":1}
I20260806 02:51:08.755769 1865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:51:08.757602 23483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 713f4e35058648d28323ee60f8a44908: Bootstrap starting.
I20260806 02:51:08.761616 23483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 713f4e35058648d28323ee60f8a44908: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:08.763064 23483 log.cc:824] T 00000000000000000000000000000000 P 713f4e35058648d28323ee60f8a44908: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:08.766274 23483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 713f4e35058648d28323ee60f8a44908: No bootstrap required, opened a new log
I20260806 02:51:08.768033 23486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:08.760483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:37947 (local address 127.0.0.1:41275)
0806 02:51:08.760717 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:08.760724 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:08.761135 (+ 411us) server_negotiation.cc:407] Connection header received
0806 02:51:08.761315 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:08.761319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:08.761372 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:08.761490 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:08.767209 (+ 5719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:08.767214 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:08.767218 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:51:08.767248 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:08.767295 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:51:08.767311 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:08.767424 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:08.767603 (+ 179us) server_negotiation.cc:299] Negotiation successful
0806 02:51:08.767663 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 02:51:08.775846 23483 raft_consensus.cc:348] T 00000000000000000000000000000000 P 713f4e35058648d28323ee60f8a44908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "713f4e35058648d28323ee60f8a44908" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41275 } }
I20260806 02:51:08.776321 23483 raft_consensus.cc:374] T 00000000000000000000000000000000 P 713f4e35058648d28323ee60f8a44908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:08.776458 23483 raft_consensus.cc:729] T 00000000000000000000000000000000 P 713f4e35058648d28323ee60f8a44908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 713f4e35058648d28323ee60f8a44908, State: Initialized, Role: FOLLOWER
I20260806 02:51:08.777163 23483 consensus_queue.cc:260] T 00000000000000000000000000000000 P 713f4e35058648d28323ee60f8a44908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "713f4e35058648d28323ee60f8a44908" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41275 } }
I20260806 02:51:08.777499 23483 raft_consensus.cc:388] T 00000000000000000000000000000000 P 713f4e35058648d28323ee60f8a44908 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:51:08.777731 23483 raft_consensus.cc:482] T 00000000000000000000000000000000 P 713f4e35058648d28323ee60f8a44908 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:51:08.777959 23483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 713f4e35058648d28323ee60f8a44908 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:08.780591 23483 raft_consensus.cc:504] T 00000000000000000000000000000000 P 713f4e35058648d28323ee60f8a44908 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "713f4e35058648d28323ee60f8a44908" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41275 } }
I20260806 02:51:08.781175 23483 leader_election.cc:302] T 00000000000000000000000000000000 P 713f4e35058648d28323ee60f8a44908 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 713f4e35058648d28323ee60f8a44908; no voters:
I20260806 02:51:08.781854 23483 leader_election.cc:288] T 00000000000000000000000000000000 P 713f4e35058648d28323ee60f8a44908 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:51:08.782116 23493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 713f4e35058648d28323ee60f8a44908 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:51:08.782940 23493 raft_consensus.cc:686] T 00000000000000000000000000000000 P 713f4e35058648d28323ee60f8a44908 [term 1 LEADER]: Becoming Leader. State: Replica: 713f4e35058648d28323ee60f8a44908, State: Running, Role: LEADER
I20260806 02:51:08.783684 23493 consensus_queue.cc:237] T 00000000000000000000000000000000 P 713f4e35058648d28323ee60f8a44908 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "713f4e35058648d28323ee60f8a44908" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41275 } }
I20260806 02:51:08.784439 23483 sys_catalog.cc:564] T 00000000000000000000000000000000 P 713f4e35058648d28323ee60f8a44908 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:51:08.787140 23494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 713f4e35058648d28323ee60f8a44908 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "713f4e35058648d28323ee60f8a44908" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "713f4e35058648d28323ee60f8a44908" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41275 } } }
I20260806 02:51:08.787402 23495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 713f4e35058648d28323ee60f8a44908 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 713f4e35058648d28323ee60f8a44908. Latest consensus state: current_term: 1 leader_uuid: "713f4e35058648d28323ee60f8a44908" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "713f4e35058648d28323ee60f8a44908" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41275 } } }
I20260806 02:51:08.787690 23494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 713f4e35058648d28323ee60f8a44908 [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:08.787700 23495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 713f4e35058648d28323ee60f8a44908 [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:08.789705 23501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:51:08.796010 23501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:51:08.806526 23501 catalog_manager.cc:1347] Generated new cluster ID: 4fc1b8f52a67471aa5ff5b90cd4258ce
I20260806 02:51:08.806666 23501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:51:08.829361 23501 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:51:08.829962 23501 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:51:08.839777 23501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 713f4e35058648d28323ee60f8a44908: Generated new TSK 0
I20260806 02:51:08.840687 23501 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:51:08.857501 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 23416
W20260806 02:51:08.859189 23478 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.51:54649
W20260806 02:51:08.860124 1865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 23416: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:51:08.860325 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 23416
2026-08-06T02:51:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2096 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:51:08 dist-test-slave-8tdl krb5kdc[23514](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:51:08 dist-test-slave-8tdl krb5kdc[23514](info): set up 2 sockets
Aug 06 02:51:08 dist-test-slave-8tdl krb5kdc[23514](info): commencing operation
krb5kdc: starting...
W20260806 02:51:10.030738 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.004s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:51:10 dist-test-slave-8tdl krb5kdc[23514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:51:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:51:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:51:10.373798 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39913
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:10.728173 23528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:10.728545 23528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:10.728724 23528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:10.736331 23528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:51:10.736550 23528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:10.736699 23528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:10.736799 23528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:51:10.736881 23528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:51:10.747395 23528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39913
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:51:10.748893 23528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:10.750360 23528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:10.763444 23533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:10.763393 23536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:10.763363 23534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:10.769805 23528 server_base.cc:1034] running on GCE node
I20260806 02:51:10.770972 23528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:10.772552 23528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:10.773783 23528 hybrid_clock.cc:648] HybridClock initialized: now 1785984670773734 us; error 39 us; skew 500 ppm
I20260806 02:51:10.774212 23528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:51:10 dist-test-slave-8tdl krb5kdc[23514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:10.781044 23528 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:51:10.782830 23528 webserver.cc:460] Webserver started at http://127.0.0.1:37151/ using document root <none> and password file <none>
I20260806 02:51:10.783612 23528 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:10.783768 23528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:10.784108 23528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:10.787353 23528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "455a49622e664444842fb9368f0c570b"
format_stamp: "Formatted at 2026-08-06 02:51:10 on dist-test-slave-8tdl"
server_key: "ff64a2cbac681bfdb331241715eaddb9"
server_key_iv: "4448d12904a231da2f4677528c93e994"
server_key_version: "encryptionkey@0"
I20260806 02:51:10.788189 23528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "455a49622e664444842fb9368f0c570b"
format_stamp: "Formatted at 2026-08-06 02:51:10 on dist-test-slave-8tdl"
server_key: "ff64a2cbac681bfdb331241715eaddb9"
server_key_iv: "4448d12904a231da2f4677528c93e994"
server_key_version: "encryptionkey@0"
I20260806 02:51:10.793776 23528 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 02:51:10.797694 23544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:10.799223 23528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:51:10.799422 23528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "455a49622e664444842fb9368f0c570b"
format_stamp: "Formatted at 2026-08-06 02:51:10 on dist-test-slave-8tdl"
server_key: "ff64a2cbac681bfdb331241715eaddb9"
server_key_iv: "4448d12904a231da2f4677528c93e994"
server_key_version: "encryptionkey@0"
I20260806 02:51:10.799670 23528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:10.815881 23528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:10.820317 23528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:10.820621 23528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:10.840657 23528 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39913
I20260806 02:51:10.840682 23596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39913 every 8 connection(s)
I20260806 02:51:10.842141 23528 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:51:10.846139 23597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:10.847219 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 23528
I20260806 02:51:10.847416 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:51:10.847908 1865 external_mini_cluster.cc:1428] Setting key d54e88e1864231d7991b0e3d3fc0f793
I20260806 02:51:10.861310 23597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 455a49622e664444842fb9368f0c570b: Bootstrap starting.
Aug 06 02:51:10 dist-test-slave-8tdl krb5kdc[23514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 02:51:10.864778 23597 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 455a49622e664444842fb9368f0c570b: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:10.866286 23597 log.cc:824] T 00000000000000000000000000000000 P 455a49622e664444842fb9368f0c570b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:10.869967 23597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 455a49622e664444842fb9368f0c570b: No bootstrap required, opened a new log
I20260806 02:51:10.873890 23600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:10.849832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:53591 (local address 127.0.0.1:39913)
0806 02:51:10.850740 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:10.850766 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:10.850819 (+ 53us) server_negotiation.cc:407] Connection header received
0806 02:51:10.853208 (+ 2389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:10.853238 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:10.853835 (+ 597us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:10.854309 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:10.855858 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:10.857957 (+ 2099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:10.859325 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:10.860410 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:10.860603 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:10.865063 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:10.865076 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:10.865095 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:10.865152 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:10.869320 (+ 4168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:10.870220 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:10.870226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:10.870233 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:10.870409 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:10.871019 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:10.871024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:10.871027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:10.871597 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:10.871857 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:10.872435 (+ 578us) server_negotiation.cc:299] Negotiation successful
0806 02:51:10.872772 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":360,"threads_started":1}
I20260806 02:51:10.877965 1865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:51:10.879290 23597 raft_consensus.cc:348] T 00000000000000000000000000000000 P 455a49622e664444842fb9368f0c570b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "455a49622e664444842fb9368f0c570b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39913 } }
I20260806 02:51:10.879740 23597 raft_consensus.cc:374] T 00000000000000000000000000000000 P 455a49622e664444842fb9368f0c570b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:10.879843 23597 raft_consensus.cc:729] T 00000000000000000000000000000000 P 455a49622e664444842fb9368f0c570b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 455a49622e664444842fb9368f0c570b, State: Initialized, Role: FOLLOWER
I20260806 02:51:10.880510 23597 consensus_queue.cc:260] T 00000000000000000000000000000000 P 455a49622e664444842fb9368f0c570b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "455a49622e664444842fb9368f0c570b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39913 } }
I20260806 02:51:10.880807 23597 raft_consensus.cc:388] T 00000000000000000000000000000000 P 455a49622e664444842fb9368f0c570b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:51:10.880954 23597 raft_consensus.cc:482] T 00000000000000000000000000000000 P 455a49622e664444842fb9368f0c570b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:51:10.881134 23597 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 455a49622e664444842fb9368f0c570b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:10.884353 23597 raft_consensus.cc:504] T 00000000000000000000000000000000 P 455a49622e664444842fb9368f0c570b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "455a49622e664444842fb9368f0c570b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39913 } }
I20260806 02:51:10.884927 23597 leader_election.cc:302] T 00000000000000000000000000000000 P 455a49622e664444842fb9368f0c570b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 455a49622e664444842fb9368f0c570b; no voters:
I20260806 02:51:10.885601 23597 leader_election.cc:288] T 00000000000000000000000000000000 P 455a49622e664444842fb9368f0c570b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:51:10.885732 23607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 455a49622e664444842fb9368f0c570b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:51:10.887916 23597 sys_catalog.cc:564] T 00000000000000000000000000000000 P 455a49622e664444842fb9368f0c570b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:51:10.892867 23607 raft_consensus.cc:686] T 00000000000000000000000000000000 P 455a49622e664444842fb9368f0c570b [term 1 LEADER]: Becoming Leader. State: Replica: 455a49622e664444842fb9368f0c570b, State: Running, Role: LEADER
I20260806 02:51:10.893656 23607 consensus_queue.cc:237] T 00000000000000000000000000000000 P 455a49622e664444842fb9368f0c570b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "455a49622e664444842fb9368f0c570b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39913 } }
I20260806 02:51:10.896996 23608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 455a49622e664444842fb9368f0c570b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "455a49622e664444842fb9368f0c570b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "455a49622e664444842fb9368f0c570b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39913 } } }
I20260806 02:51:10.897647 23609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 455a49622e664444842fb9368f0c570b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 455a49622e664444842fb9368f0c570b. Latest consensus state: current_term: 1 leader_uuid: "455a49622e664444842fb9368f0c570b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "455a49622e664444842fb9368f0c570b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39913 } } }
I20260806 02:51:10.898080 23609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 455a49622e664444842fb9368f0c570b [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:10.899367 23608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 455a49622e664444842fb9368f0c570b [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:10.902151 23614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:51:10.906976 23600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:10.883369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:43533 (local address 127.0.0.1:39913)
0806 02:51:10.883576 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:10.883582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:10.884104 (+ 522us) server_negotiation.cc:407] Connection header received
0806 02:51:10.884260 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:10.884268 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:10.884363 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:10.884475 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:10.886495 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:10.888084 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:10.890755 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:10.891905 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:10.892108 (+ 203us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:10.895969 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:10.895977 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:10.895982 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:10.896030 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:10.901877 (+ 5847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:10.903563 (+ 1686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:10.903571 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:10.903574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:10.903788 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:10.904539 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:10.904548 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:10.904554 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:10.905163 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:10.905557 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:10.906127 (+ 570us) server_negotiation.cc:299] Negotiation successful
0806 02:51:10.906416 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 02:51:10.914695 23614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:51:10.924813 23614 catalog_manager.cc:1347] Generated new cluster ID: 890e3c513f26482b8d389669ede2ba70
I20260806 02:51:10.924942 23614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:51:10.933362 23614 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:51:10.935006 23614 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:51:10.950431 23614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 455a49622e664444842fb9368f0c570b: Generated new TSK 0
I20260806 02:51:10.951465 23614 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:51:10.967445 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 23528
I20260806 02:51:10.968808 23592 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:51:11.064195 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 23528
W20260806 02:51:11.086330 23512 connection.cc:535] client connection to 127.0.0.1:39913 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39913): Connection reset by peer (error 104)
2026-08-06T02:51:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2204 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:51:11 dist-test-slave-8tdl krb5kdc[23629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:51:11 dist-test-slave-8tdl krb5kdc[23629](info): set up 2 sockets
Aug 06 02:51:11 dist-test-slave-8tdl krb5kdc[23629](info): commencing operation
krb5kdc: starting...
W20260806 02:51:12.246140 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.000s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:51:12 dist-test-slave-8tdl krb5kdc[23629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:51:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:51:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:51:12.587550 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38443
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:12.946131 23642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:12.946509 23642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:12.946707 23642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:12.954361 23642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:51:12.954542 23642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:12.954648 23642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:12.954772 23642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:51:12.954849 23642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:51:12.965464 23642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38443
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:51:12.967002 23642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:12.968343 23642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:12.980806 23648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:12.981005 23647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:12.981061 23650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:12.981669 23642 server_base.cc:1034] running on GCE node
I20260806 02:51:12.982539 23642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:12.984201 23642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:12.985452 23642 hybrid_clock.cc:648] HybridClock initialized: now 1785984672985407 us; error 46 us; skew 500 ppm
I20260806 02:51:12.985872 23642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:51:12 dist-test-slave-8tdl krb5kdc[23629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:12.989339 23642 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:51:12.991097 23642 webserver.cc:460] Webserver started at http://127.0.0.1:36239/ using document root <none> and password file <none>
I20260806 02:51:12.991832 23642 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:12.991989 23642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:12.992312 23642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:12.995671 23642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "f29f19a723fe41a188fd48f6027532a5"
format_stamp: "Formatted at 2026-08-06 02:51:12 on dist-test-slave-8tdl"
server_key: "29b9aac9867675863b9a5f520a3a583c"
server_key_iv: "4c2c240595b494c251a5c54ce73d078c"
server_key_version: "encryptionkey@0"
I20260806 02:51:12.996515 23642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "f29f19a723fe41a188fd48f6027532a5"
format_stamp: "Formatted at 2026-08-06 02:51:12 on dist-test-slave-8tdl"
server_key: "29b9aac9867675863b9a5f520a3a583c"
server_key_iv: "4c2c240595b494c251a5c54ce73d078c"
server_key_version: "encryptionkey@0"
I20260806 02:51:13.002362 23642 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260806 02:51:13.006790 23657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:13.008741 23642 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260806 02:51:13.008989 23642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "f29f19a723fe41a188fd48f6027532a5"
format_stamp: "Formatted at 2026-08-06 02:51:12 on dist-test-slave-8tdl"
server_key: "29b9aac9867675863b9a5f520a3a583c"
server_key_iv: "4c2c240595b494c251a5c54ce73d078c"
server_key_version: "encryptionkey@0"
I20260806 02:51:13.009246 23642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:13.023483 23642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:13.028223 23642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:13.028568 23642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:13.049638 23642 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38443
I20260806 02:51:13.049667 23710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38443 every 8 connection(s)
I20260806 02:51:13.051328 23642 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:51:13.055338 23711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:13.061365 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 23642
I20260806 02:51:13.061575 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:51:13.062142 1865 external_mini_cluster.cc:1428] Setting key 039380e3ac5c5fac11b0757820107216
I20260806 02:51:13.068835 23711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f29f19a723fe41a188fd48f6027532a5: Bootstrap starting.
I20260806 02:51:13.073321 23711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f29f19a723fe41a188fd48f6027532a5: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:13.074981 23711 log.cc:824] T 00000000000000000000000000000000 P f29f19a723fe41a188fd48f6027532a5: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:13.079007 23711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f29f19a723fe41a188fd48f6027532a5: No bootstrap required, opened a new log
I20260806 02:51:13.086234 23714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:13.070642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:57503 (local address 127.0.0.1:38443)
0806 02:51:13.071629 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:13.071643 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:13.071686 (+ 43us) server_negotiation.cc:407] Connection header received
0806 02:51:13.074482 (+ 2796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:13.074518 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:13.075178 (+ 660us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:13.075699 (+ 521us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:13.076864 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:13.078998 (+ 2134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:13.080371 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:13.081490 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:13.081690 (+ 200us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:13.083188 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:13.083203 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:13.083223 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:51:13.083989 (+ 766us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:13.084098 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:51:13.084138 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:13.084313 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:13.084588 (+ 275us) server_negotiation.cc:299] Negotiation successful
0806 02:51:13.084798 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":433,"threads_started":1}
I20260806 02:51:13.090058 23711 raft_consensus.cc:348] T 00000000000000000000000000000000 P f29f19a723fe41a188fd48f6027532a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29f19a723fe41a188fd48f6027532a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38443 } }
I20260806 02:51:13.090476 23711 raft_consensus.cc:374] T 00000000000000000000000000000000 P f29f19a723fe41a188fd48f6027532a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:13.090611 23711 raft_consensus.cc:729] T 00000000000000000000000000000000 P f29f19a723fe41a188fd48f6027532a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f29f19a723fe41a188fd48f6027532a5, State: Initialized, Role: FOLLOWER
I20260806 02:51:13.091127 1865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:51:13.091280 23711 consensus_queue.cc:260] T 00000000000000000000000000000000 P f29f19a723fe41a188fd48f6027532a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29f19a723fe41a188fd48f6027532a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38443 } }
I20260806 02:51:13.091576 23711 raft_consensus.cc:388] T 00000000000000000000000000000000 P f29f19a723fe41a188fd48f6027532a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:51:13.091706 23711 raft_consensus.cc:482] T 00000000000000000000000000000000 P f29f19a723fe41a188fd48f6027532a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:51:13.091873 23711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f29f19a723fe41a188fd48f6027532a5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:13.095422 23711 raft_consensus.cc:504] T 00000000000000000000000000000000 P f29f19a723fe41a188fd48f6027532a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29f19a723fe41a188fd48f6027532a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38443 } }
I20260806 02:51:13.096118 23711 leader_election.cc:302] T 00000000000000000000000000000000 P f29f19a723fe41a188fd48f6027532a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f29f19a723fe41a188fd48f6027532a5; no voters:
I20260806 02:51:13.096827 23711 leader_election.cc:288] T 00000000000000000000000000000000 P f29f19a723fe41a188fd48f6027532a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:51:13.097105 23720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f29f19a723fe41a188fd48f6027532a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:51:13.100152 23720 raft_consensus.cc:686] T 00000000000000000000000000000000 P f29f19a723fe41a188fd48f6027532a5 [term 1 LEADER]: Becoming Leader. State: Replica: f29f19a723fe41a188fd48f6027532a5, State: Running, Role: LEADER
I20260806 02:51:13.100900 23720 consensus_queue.cc:237] T 00000000000000000000000000000000 P f29f19a723fe41a188fd48f6027532a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29f19a723fe41a188fd48f6027532a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38443 } }
I20260806 02:51:13.101075 23711 sys_catalog.cc:564] T 00000000000000000000000000000000 P f29f19a723fe41a188fd48f6027532a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:51:13.104245 23723 sys_catalog.cc:455] T 00000000000000000000000000000000 P f29f19a723fe41a188fd48f6027532a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f29f19a723fe41a188fd48f6027532a5. Latest consensus state: current_term: 1 leader_uuid: "f29f19a723fe41a188fd48f6027532a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29f19a723fe41a188fd48f6027532a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38443 } } }
I20260806 02:51:13.104694 23723 sys_catalog.cc:458] T 00000000000000000000000000000000 P f29f19a723fe41a188fd48f6027532a5 [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:13.105309 23722 sys_catalog.cc:455] T 00000000000000000000000000000000 P f29f19a723fe41a188fd48f6027532a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f29f19a723fe41a188fd48f6027532a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f29f19a723fe41a188fd48f6027532a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38443 } } }
I20260806 02:51:13.105688 23722 sys_catalog.cc:458] T 00000000000000000000000000000000 P f29f19a723fe41a188fd48f6027532a5 [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:13.110606 23714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:13.096967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:40959 (local address 127.0.0.1:38443)
0806 02:51:13.097191 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:13.097218 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:13.099935 (+ 2717us) server_negotiation.cc:407] Connection header received
0806 02:51:13.100450 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:13.100456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:13.100541 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:13.100630 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:13.101582 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:13.103080 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:13.105770 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:13.107062 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:13.108179 (+ 1117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:13.109071 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:13.109081 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:13.109089 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:51:13.109120 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:13.109274 (+ 154us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:51:13.109302 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:13.109433 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:13.110040 (+ 607us) server_negotiation.cc:299] Negotiation successful
0806 02:51:13.110114 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 02:51:13.114970 23727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:51:13.121622 23727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:51:13.133078 23727 catalog_manager.cc:1347] Generated new cluster ID: 36a0439c657f483d9a32ac38c4e2d236
I20260806 02:51:13.133230 23727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:51:13.164320 23727 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:51:13.166608 23727 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:51:13.178879 23727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f29f19a723fe41a188fd48f6027532a5: Generated new TSK 0
I20260806 02:51:13.180109 23727 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:51:13.190146 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 23642
W20260806 02:51:13.191730 23706 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.51:57503
W20260806 02:51:13.192643 1865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 23642: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:51:13.192826 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 23642
W20260806 02:51:13.213706 23627 connection.cc:535] client connection to 127.0.0.1:38443 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38443): Connection reset by peer (error 104)
2026-08-06T02:51:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2127 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:51:13 dist-test-slave-8tdl krb5kdc[23742](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:51:13 dist-test-slave-8tdl krb5kdc[23742](info): set up 2 sockets
Aug 06 02:51:13 dist-test-slave-8tdl krb5kdc[23742](info): commencing operation
krb5kdc: starting...
W20260806 02:51:14.350440 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:51:14 dist-test-slave-8tdl krb5kdc[23742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:51:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:51:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:51:14.678833 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34945
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:15.034418 23755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:15.034819 23755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:15.034967 23755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:15.042587 23755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:51:15.042791 23755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:15.042862 23755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:15.042946 23755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:51:15.042994 23755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:51:15.053541 23755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34945
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984304136196-1865-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:51:15.054919 23755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:15.056319 23755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:15.070075 23761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:15.070152 23763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:15.070173 23760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:15.076556 23755 server_base.cc:1034] running on GCE node
I20260806 02:51:15.077467 23755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:15.079151 23755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:15.080376 23755 hybrid_clock.cc:648] HybridClock initialized: now 1785984675080323 us; error 44 us; skew 500 ppm
I20260806 02:51:15.080861 23755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:51:15 dist-test-slave-8tdl krb5kdc[23742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:15.085209 23755 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:51:15.087019 23755 webserver.cc:460] Webserver started at http://127.0.0.1:45075/ using document root <none> and password file <none>
I20260806 02:51:15.087807 23755 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:15.087965 23755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:15.088300 23755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:15.091817 23755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "bccc87335e50416e9dfd4840507317ad"
format_stamp: "Formatted at 2026-08-06 02:51:15 on dist-test-slave-8tdl"
server_key: "fa7c1951a2c90e392008c06d10f189e5"
server_key_iv: "c48b62037a263b5904e116b5ac48c4dd"
server_key_version: "encryptionkey@0"
I20260806 02:51:15.092715 23755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "bccc87335e50416e9dfd4840507317ad"
format_stamp: "Formatted at 2026-08-06 02:51:15 on dist-test-slave-8tdl"
server_key: "fa7c1951a2c90e392008c06d10f189e5"
server_key_iv: "c48b62037a263b5904e116b5ac48c4dd"
server_key_version: "encryptionkey@0"
I20260806 02:51:15.098543 23755 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 02:51:15.102840 23770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:15.104481 23755 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260806 02:51:15.104717 23755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "bccc87335e50416e9dfd4840507317ad"
format_stamp: "Formatted at 2026-08-06 02:51:15 on dist-test-slave-8tdl"
server_key: "fa7c1951a2c90e392008c06d10f189e5"
server_key_iv: "c48b62037a263b5904e116b5ac48c4dd"
server_key_version: "encryptionkey@0"
I20260806 02:51:15.105031 23755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:15.112195 23755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:15.117479 23755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:15.117848 23755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:15.139519 23755 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34945
I20260806 02:51:15.139570 23821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34945 every 8 connection(s)
I20260806 02:51:15.141034 23755 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:51:15.142590 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 23755
I20260806 02:51:15.142820 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:51:15.143308 1865 external_mini_cluster.cc:1428] Setting key d056337b88e324130a22ea473adba3cf
I20260806 02:51:15.145646 23822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:15.151693 23825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:15.145125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:52859 (local address 127.0.0.1:34945)
0806 02:51:15.146088 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:15.146101 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:15.146146 (+ 45us) server_negotiation.cc:407] Connection header received
0806 02:51:15.147924 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:15.147947 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:15.148318 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:15.148753 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:15.149875 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:15.149881 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:15.149897 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:51:15.149912 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:15.149988 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:51:15.150013 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:15.150315 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:15.150470 (+ 155us) server_negotiation.cc:299] Negotiation successful
0806 02:51:15.150717 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":412,"threads_started":1}
I20260806 02:51:15.156102 1865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:51:15.158329 23822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bccc87335e50416e9dfd4840507317ad: Bootstrap starting.
I20260806 02:51:15.161972 23822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bccc87335e50416e9dfd4840507317ad: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:15.163580 23825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:15.160748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:58583 (local address 127.0.0.1:34945)
0806 02:51:15.160947 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:15.160953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:15.161493 (+ 540us) server_negotiation.cc:407] Connection header received
0806 02:51:15.161639 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:15.161644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:15.161712 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:15.161838 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:15.162881 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:15.162887 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:15.162891 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:51:15.162910 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:15.162958 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:51:15.162979 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:15.163105 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:15.163202 (+ 97us) server_negotiation.cc:299] Negotiation successful
0806 02:51:15.163274 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 02:51:15.163516 23822 log.cc:824] T 00000000000000000000000000000000 P bccc87335e50416e9dfd4840507317ad: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:15.166956 23822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bccc87335e50416e9dfd4840507317ad: No bootstrap required, opened a new log
I20260806 02:51:15.174466 23822 raft_consensus.cc:348] T 00000000000000000000000000000000 P bccc87335e50416e9dfd4840507317ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bccc87335e50416e9dfd4840507317ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34945 } }
I20260806 02:51:15.174979 23822 raft_consensus.cc:374] T 00000000000000000000000000000000 P bccc87335e50416e9dfd4840507317ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:15.175105 23822 raft_consensus.cc:729] T 00000000000000000000000000000000 P bccc87335e50416e9dfd4840507317ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bccc87335e50416e9dfd4840507317ad, State: Initialized, Role: FOLLOWER
I20260806 02:51:15.175729 23822 consensus_queue.cc:260] T 00000000000000000000000000000000 P bccc87335e50416e9dfd4840507317ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bccc87335e50416e9dfd4840507317ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34945 } }
I20260806 02:51:15.175966 23822 raft_consensus.cc:388] T 00000000000000000000000000000000 P bccc87335e50416e9dfd4840507317ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:51:15.176138 23822 raft_consensus.cc:482] T 00000000000000000000000000000000 P bccc87335e50416e9dfd4840507317ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:51:15.176343 23822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bccc87335e50416e9dfd4840507317ad [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:15.178714 23822 raft_consensus.cc:504] T 00000000000000000000000000000000 P bccc87335e50416e9dfd4840507317ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bccc87335e50416e9dfd4840507317ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34945 } }
I20260806 02:51:15.179301 23822 leader_election.cc:302] T 00000000000000000000000000000000 P bccc87335e50416e9dfd4840507317ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bccc87335e50416e9dfd4840507317ad; no voters:
I20260806 02:51:15.179955 23822 leader_election.cc:288] T 00000000000000000000000000000000 P bccc87335e50416e9dfd4840507317ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:51:15.180135 23833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bccc87335e50416e9dfd4840507317ad [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:51:15.181033 23833 raft_consensus.cc:686] T 00000000000000000000000000000000 P bccc87335e50416e9dfd4840507317ad [term 1 LEADER]: Becoming Leader. State: Replica: bccc87335e50416e9dfd4840507317ad, State: Running, Role: LEADER
I20260806 02:51:15.181639 23833 consensus_queue.cc:237] T 00000000000000000000000000000000 P bccc87335e50416e9dfd4840507317ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bccc87335e50416e9dfd4840507317ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34945 } }
I20260806 02:51:15.182346 23822 sys_catalog.cc:564] T 00000000000000000000000000000000 P bccc87335e50416e9dfd4840507317ad [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:51:15.184907 23835 sys_catalog.cc:455] T 00000000000000000000000000000000 P bccc87335e50416e9dfd4840507317ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader bccc87335e50416e9dfd4840507317ad. Latest consensus state: current_term: 1 leader_uuid: "bccc87335e50416e9dfd4840507317ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bccc87335e50416e9dfd4840507317ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34945 } } }
I20260806 02:51:15.185272 23835 sys_catalog.cc:458] T 00000000000000000000000000000000 P bccc87335e50416e9dfd4840507317ad [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:15.186233 23840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:51:15.186034 23834 sys_catalog.cc:455] T 00000000000000000000000000000000 P bccc87335e50416e9dfd4840507317ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bccc87335e50416e9dfd4840507317ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bccc87335e50416e9dfd4840507317ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34945 } } }
I20260806 02:51:15.187734 23834 sys_catalog.cc:458] T 00000000000000000000000000000000 P bccc87335e50416e9dfd4840507317ad [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:15.192236 23840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:51:15.202133 23840 catalog_manager.cc:1347] Generated new cluster ID: 7ca060a94c214cb4ad017c84a0f08bcf
I20260806 02:51:15.202404 23840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:51:15.225418 23840 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:51:15.225934 23840 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:51:15.243785 23840 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bccc87335e50416e9dfd4840507317ad: Generated new TSK 0
I20260806 02:51:15.244700 23840 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:51:15.251639 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 23755
W20260806 02:51:15.253027 23817 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.51:52859
W20260806 02:51:15.253809 1865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 23755: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:51:15.253978 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 23755
2026-08-06T02:51:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2060 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:51:15 dist-test-slave-8tdl krb5kdc[23854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:51:15 dist-test-slave-8tdl krb5kdc[23854](info): set up 2 sockets
Aug 06 02:51:15 dist-test-slave-8tdl krb5kdc[23854](info): commencing operation
krb5kdc: starting...
W20260806 02:51:16.414619 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.000s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:51:16 dist-test-slave-8tdl krb5kdc[23854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:51:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:51:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
I20260806 02:51:16.753811 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:45943
--webserver_interface=127.1.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:34245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:45943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:17.118613 23867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:17.119025 23867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:17.119242 23867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:17.128139 23867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:51:17.128288 23867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:17.128376 23867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:17.128492 23867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:51:17.128561 23867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:51:17.139566 23867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:34245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:45943
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:45943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:51:17.141108 23867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:17.142532 23867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:17.155941 23873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:17.155828 23875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:17.155840 23872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:17.157481 23867 server_base.cc:1034] running on GCE node
I20260806 02:51:17.158460 23867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:17.160212 23867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:17.161468 23867 hybrid_clock.cc:648] HybridClock initialized: now 1785984677161409 us; error 64 us; skew 500 ppm
I20260806 02:51:17.161875 23867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:51:17 dist-test-slave-8tdl krb5kdc[23854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984677, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:17.170943 23867 init.cc:377] Logged in from keytab as kudu/127.1.210.126@KRBTEST.COM (short username kudu)
I20260806 02:51:17.172680 23867 webserver.cc:460] Webserver started at http://127.1.210.126:45277/ using document root <none> and password file <none>
I20260806 02:51:17.173439 23867 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:17.173602 23867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:17.173949 23867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:17.177353 23867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "9ecfd8aa17394f889202aa56f9bba832"
format_stamp: "Formatted at 2026-08-06 02:51:17 on dist-test-slave-8tdl"
server_key: "a4bc81df77b2ab20876048ae5724465a"
server_key_iv: "db8636162b6d1dc237234831cdaa9e8a"
server_key_version: "encryptionkey@0"
I20260806 02:51:17.178263 23867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "9ecfd8aa17394f889202aa56f9bba832"
format_stamp: "Formatted at 2026-08-06 02:51:17 on dist-test-slave-8tdl"
server_key: "a4bc81df77b2ab20876048ae5724465a"
server_key_iv: "db8636162b6d1dc237234831cdaa9e8a"
server_key_version: "encryptionkey@0"
I20260806 02:51:17.184278 23867 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260806 02:51:17.188426 23882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:17.190217 23867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:51:17.190431 23867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "9ecfd8aa17394f889202aa56f9bba832"
format_stamp: "Formatted at 2026-08-06 02:51:17 on dist-test-slave-8tdl"
server_key: "a4bc81df77b2ab20876048ae5724465a"
server_key_iv: "db8636162b6d1dc237234831cdaa9e8a"
server_key_version: "encryptionkey@0"
I20260806 02:51:17.190716 23867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:17.210052 23867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:17.214967 23867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:17.215287 23867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:17.236449 23867 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:45943
I20260806 02:51:17.236477 23933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:45943 every 8 connection(s)
I20260806 02:51:17.238159 23867 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:51:17.242435 23934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:17.247886 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 23867
I20260806 02:51:17.248144 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:51:17.248620 1865 external_mini_cluster.cc:1428] Setting key 8e96abf55d98810aad4a62847d0e6c70
I20260806 02:51:17.254465 23934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ecfd8aa17394f889202aa56f9bba832: Bootstrap starting.
I20260806 02:51:17.257889 23934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ecfd8aa17394f889202aa56f9bba832: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:17.259085 23934 log.cc:824] T 00000000000000000000000000000000 P 9ecfd8aa17394f889202aa56f9bba832: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:17.262156 23934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ecfd8aa17394f889202aa56f9bba832: No bootstrap required, opened a new log
Aug 06 02:51:17 dist-test-slave-8tdl krb5kdc[23854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:51:17.271920 23934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ecfd8aa17394f889202aa56f9bba832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ecfd8aa17394f889202aa56f9bba832" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45943 } }
I20260806 02:51:17.272344 23934 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ecfd8aa17394f889202aa56f9bba832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:17.272495 23934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ecfd8aa17394f889202aa56f9bba832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ecfd8aa17394f889202aa56f9bba832, State: Initialized, Role: FOLLOWER
I20260806 02:51:17.273275 23937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:17.250631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36411 (local address 127.1.210.126:45943)
0806 02:51:17.251626 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:17.251640 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:17.251688 (+ 48us) server_negotiation.cc:407] Connection header received
0806 02:51:17.253548 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:17.253573 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:17.254019 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:17.254560 (+ 541us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:17.256315 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:17.257673 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:17.259094 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:17.259931 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:17.260178 (+ 247us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:17.264977 (+ 4799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:17.264985 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:17.265002 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:17.265040 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:17.268453 (+ 3413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:17.269429 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:17.269435 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:17.269442 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:17.269622 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:17.270108 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:17.270117 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:17.270119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:17.270789 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:17.271051 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:17.271556 (+ 505us) server_negotiation.cc:299] Negotiation successful
0806 02:51:17.271906 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":460,"threads_started":1}
I20260806 02:51:17.273180 23934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ecfd8aa17394f889202aa56f9bba832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ecfd8aa17394f889202aa56f9bba832" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45943 } }
I20260806 02:51:17.273464 23934 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ecfd8aa17394f889202aa56f9bba832 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:51:17.273643 23934 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ecfd8aa17394f889202aa56f9bba832 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:51:17.273835 23934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ecfd8aa17394f889202aa56f9bba832 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:17.276611 23934 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ecfd8aa17394f889202aa56f9bba832 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ecfd8aa17394f889202aa56f9bba832" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45943 } }
I20260806 02:51:17.277158 23934 leader_election.cc:302] T 00000000000000000000000000000000 P 9ecfd8aa17394f889202aa56f9bba832 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ecfd8aa17394f889202aa56f9bba832; no voters:
I20260806 02:51:17.277519 1865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:51:17.277908 23934 leader_election.cc:288] T 00000000000000000000000000000000 P 9ecfd8aa17394f889202aa56f9bba832 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:51:17.277989 23939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ecfd8aa17394f889202aa56f9bba832 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:51:17.279597 23939 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ecfd8aa17394f889202aa56f9bba832 [term 1 LEADER]: Becoming Leader. State: Replica: 9ecfd8aa17394f889202aa56f9bba832, State: Running, Role: LEADER
I20260806 02:51:17.280388 23934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ecfd8aa17394f889202aa56f9bba832 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:51:17.280301 23939 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ecfd8aa17394f889202aa56f9bba832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ecfd8aa17394f889202aa56f9bba832" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45943 } }
I20260806 02:51:17.291499 23941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ecfd8aa17394f889202aa56f9bba832 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ecfd8aa17394f889202aa56f9bba832. Latest consensus state: current_term: 1 leader_uuid: "9ecfd8aa17394f889202aa56f9bba832" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ecfd8aa17394f889202aa56f9bba832" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45943 } } }
I20260806 02:51:17.292038 23941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ecfd8aa17394f889202aa56f9bba832 [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:17.293059 23940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ecfd8aa17394f889202aa56f9bba832 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ecfd8aa17394f889202aa56f9bba832" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ecfd8aa17394f889202aa56f9bba832" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45943 } } }
I20260806 02:51:17.293464 23940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ecfd8aa17394f889202aa56f9bba832 [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:17.299073 23955 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:51:17.305948 23955 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:51:17.312728 23937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:17.290241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49031 (local address 127.1.210.126:45943)
0806 02:51:17.290505 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:17.290515 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:17.291992 (+ 1477us) server_negotiation.cc:407] Connection header received
0806 02:51:17.292310 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:17.292318 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:17.292417 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:17.292500 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:17.295256 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:17.296812 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:17.299891 (+ 3079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:17.301403 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:17.301607 (+ 204us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:17.305071 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:17.305079 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:17.305084 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:17.305135 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:17.309385 (+ 4250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:17.310335 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:17.310340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:17.310349 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:17.310494 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:17.311070 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:17.311074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:17.311076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:17.311399 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:17.311716 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:17.312120 (+ 404us) server_negotiation.cc:299] Negotiation successful
0806 02:51:17.312324 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 02:51:17.317652 23955 catalog_manager.cc:1347] Generated new cluster ID: e35f4eaa555d4ac4a1f4da615d83e3c1
I20260806 02:51:17.317830 23955 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:51:17.347079 23955 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:51:17.348807 23955 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:51:17.361307 23955 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ecfd8aa17394f889202aa56f9bba832: Generated new TSK 0
I20260806 02:51:17.362252 23955 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:51:17.371330 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 23867
I20260806 02:51:17.372638 23929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:51:17.471021 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 23867
W20260806 02:51:17.492519 23852 connection.cc:535] client connection to 127.1.210.126:45943 recv error: Network error: failed to read from TLS socket (remote: 127.1.210.126:45943): Connection reset by peer (error 104)
2026-08-06T02:51:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2218 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:51:17 dist-test-slave-8tdl krb5kdc[23966](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:51:17 dist-test-slave-8tdl krb5kdc[23966](info): set up 2 sockets
Aug 06 02:51:17 dist-test-slave-8tdl krb5kdc[23966](info): commencing operation
krb5kdc: starting...
W20260806 02:51:18.622946 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.000s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:51:18 dist-test-slave-8tdl krb5kdc[23966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:51:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:51:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
I20260806 02:51:18.974406 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:41415
--webserver_interface=127.1.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:37787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:41415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:19.352581 23979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:19.352909 23979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:19.353089 23979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:19.361086 23979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:51:19.361220 23979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:19.361320 23979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:19.361418 23979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:51:19.361497 23979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:51:19.372053 23979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:37787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:41415
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:41415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:51:19.373514 23979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:19.374908 23979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:19.388141 23984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:19.388250 23987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:19.388221 23985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:19.389117 23979 server_base.cc:1034] running on GCE node
I20260806 02:51:19.389909 23979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:19.391590 23979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:19.392814 23979 hybrid_clock.cc:648] HybridClock initialized: now 1785984679392772 us; error 32 us; skew 500 ppm
I20260806 02:51:19.393229 23979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:51:19 dist-test-slave-8tdl krb5kdc[23966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984679, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:19.396899 23979 init.cc:377] Logged in from keytab as kudu/127.1.210.126@KRBTEST.COM (short username kudu)
I20260806 02:51:19.398763 23979 webserver.cc:460] Webserver started at http://127.1.210.126:40741/ using document root <none> and password file <none>
I20260806 02:51:19.399519 23979 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:19.399678 23979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:19.400009 23979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:19.403278 23979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "9d9a83379c604676b08954b31acc4ced"
format_stamp: "Formatted at 2026-08-06 02:51:19 on dist-test-slave-8tdl"
server_key: "2a8b2baa9227bfdd58d7d329939fc606"
server_key_iv: "1672145a8b7358bcb5424d8bcdd97510"
server_key_version: "encryptionkey@0"
I20260806 02:51:19.404111 23979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "9d9a83379c604676b08954b31acc4ced"
format_stamp: "Formatted at 2026-08-06 02:51:19 on dist-test-slave-8tdl"
server_key: "2a8b2baa9227bfdd58d7d329939fc606"
server_key_iv: "1672145a8b7358bcb5424d8bcdd97510"
server_key_version: "encryptionkey@0"
I20260806 02:51:19.409883 23979 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 02:51:19.413985 23994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:19.415702 23979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 02:51:19.415934 23979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "9d9a83379c604676b08954b31acc4ced"
format_stamp: "Formatted at 2026-08-06 02:51:19 on dist-test-slave-8tdl"
server_key: "2a8b2baa9227bfdd58d7d329939fc606"
server_key_iv: "1672145a8b7358bcb5424d8bcdd97510"
server_key_version: "encryptionkey@0"
I20260806 02:51:19.416240 23979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:19.434564 23979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:19.439605 23979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:19.439924 23979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:19.460597 23979 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:41415
I20260806 02:51:19.460628 24047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:41415 every 8 connection(s)
I20260806 02:51:19.462188 23979 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:51:19.466271 24048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:19.467373 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 23979
I20260806 02:51:19.467594 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:51:19.468079 1865 external_mini_cluster.cc:1428] Setting key 00a10180b80d95f772fdf903b9b5ec2c
I20260806 02:51:19.481513 24048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d9a83379c604676b08954b31acc4ced: Bootstrap starting.
I20260806 02:51:19.483388 24051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:19.470013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60685 (local address 127.1.210.126:41415)
0806 02:51:19.470869 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:19.470881 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:19.470927 (+ 46us) server_negotiation.cc:407] Connection header received
0806 02:51:19.472880 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:19.472911 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:19.473374 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:19.473755 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:19.474563 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:19.476039 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:19.478105 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:19.479052 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:19.479211 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:19.480663 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:19.480671 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:19.480689 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:51:19.481346 (+ 657us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:19.481435 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:51:19.481462 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:19.481598 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:19.481984 (+ 386us) server_negotiation.cc:299] Negotiation successful
0806 02:51:19.482202 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":401,"threads_started":1}
I20260806 02:51:19.485203 24048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d9a83379c604676b08954b31acc4ced: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:19.486485 24048 log.cc:824] T 00000000000000000000000000000000 P 9d9a83379c604676b08954b31acc4ced: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:19.487871 1865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:51:19.490200 24048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d9a83379c604676b08954b31acc4ced: No bootstrap required, opened a new log
I20260806 02:51:19.500978 24051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:19.493788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37957 (local address 127.1.210.126:41415)
0806 02:51:19.494006 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:19.494014 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:19.494406 (+ 392us) server_negotiation.cc:407] Connection header received
0806 02:51:19.494603 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:19.494608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:19.494726 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:19.494833 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:19.495409 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:19.496640 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:19.498029 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:19.498795 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:19.498952 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:19.500050 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:19.500057 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:19.500062 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:51:19.500096 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:19.500162 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:51:19.500188 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:19.500305 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:19.500578 (+ 273us) server_negotiation.cc:299] Negotiation successful
0806 02:51:19.500641 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 02:51:19.501114 24048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d9a83379c604676b08954b31acc4ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d9a83379c604676b08954b31acc4ced" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41415 } }
I20260806 02:51:19.501590 24048 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d9a83379c604676b08954b31acc4ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:19.501705 24048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d9a83379c604676b08954b31acc4ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d9a83379c604676b08954b31acc4ced, State: Initialized, Role: FOLLOWER
I20260806 02:51:19.502347 24048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d9a83379c604676b08954b31acc4ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d9a83379c604676b08954b31acc4ced" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41415 } }
I20260806 02:51:19.502645 24048 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d9a83379c604676b08954b31acc4ced [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:51:19.502837 24048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d9a83379c604676b08954b31acc4ced [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:51:19.503037 24048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d9a83379c604676b08954b31acc4ced [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:19.505489 24048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d9a83379c604676b08954b31acc4ced [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d9a83379c604676b08954b31acc4ced" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41415 } }
I20260806 02:51:19.506094 24048 leader_election.cc:302] T 00000000000000000000000000000000 P 9d9a83379c604676b08954b31acc4ced [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d9a83379c604676b08954b31acc4ced; no voters:
I20260806 02:51:19.506843 24048 leader_election.cc:288] T 00000000000000000000000000000000 P 9d9a83379c604676b08954b31acc4ced [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:51:19.506938 24058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d9a83379c604676b08954b31acc4ced [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:51:19.507767 24058 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d9a83379c604676b08954b31acc4ced [term 1 LEADER]: Becoming Leader. State: Replica: 9d9a83379c604676b08954b31acc4ced, State: Running, Role: LEADER
I20260806 02:51:19.508365 24058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d9a83379c604676b08954b31acc4ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d9a83379c604676b08954b31acc4ced" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41415 } }
I20260806 02:51:19.509112 24048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d9a83379c604676b08954b31acc4ced [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:51:19.513852 24060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d9a83379c604676b08954b31acc4ced [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d9a83379c604676b08954b31acc4ced. Latest consensus state: current_term: 1 leader_uuid: "9d9a83379c604676b08954b31acc4ced" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d9a83379c604676b08954b31acc4ced" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41415 } } }
I20260806 02:51:19.513813 24059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d9a83379c604676b08954b31acc4ced [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d9a83379c604676b08954b31acc4ced" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d9a83379c604676b08954b31acc4ced" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41415 } } }
I20260806 02:51:19.514971 24060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d9a83379c604676b08954b31acc4ced [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:19.515024 24059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d9a83379c604676b08954b31acc4ced [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:19.516503 24068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:51:19.523654 24068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:51:19.533811 24068 catalog_manager.cc:1347] Generated new cluster ID: c3c63d95832e4ae1b5f7a1fddec82732
I20260806 02:51:19.533959 24068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:51:19.563563 24068 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:51:19.565821 24068 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:51:19.584802 24068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d9a83379c604676b08954b31acc4ced: Generated new TSK 0
I20260806 02:51:19.585770 24068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:51:19.603430 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 23979
W20260806 02:51:19.604914 24043 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60685
W20260806 02:51:19.605815 1865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 23979: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:51:19.605991 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 23979
W20260806 02:51:19.627045 23964 connection.cc:535] client connection to 127.1.210.126:41415 recv error: Network error: failed to read from TLS socket (remote: 127.1.210.126:41415): Connection reset by peer (error 104)
2026-08-06T02:51:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2134 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:51:19 dist-test-slave-8tdl krb5kdc[24079](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:51:19 dist-test-slave-8tdl krb5kdc[24079](info): set up 2 sockets
Aug 06 02:51:19 dist-test-slave-8tdl krb5kdc[24079](info): commencing operation
krb5kdc: starting...
W20260806 02:51:20.758513 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.003s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:51:20 dist-test-slave-8tdl krb5kdc[24079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:51:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:51:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
I20260806 02:51:21.102911 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:42751
--webserver_interface=127.1.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:37141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:42751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:21.460978 24092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:21.461387 24092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:21.461580 24092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:21.469998 24092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:51:21.470181 24092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:21.470310 24092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:21.470434 24092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:51:21.470538 24092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:51:21.481912 24092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:37141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:42751
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:42751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:51:21.483415 24092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:21.484830 24092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:21.498085 24100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:21.498114 24098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:21.498201 24097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:21.499243 24092 server_base.cc:1034] running on GCE node
I20260806 02:51:21.500190 24092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:21.501791 24092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:21.503041 24092 hybrid_clock.cc:648] HybridClock initialized: now 1785984681503001 us; error 50 us; skew 500 ppm
I20260806 02:51:21.503414 24092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:51:21 dist-test-slave-8tdl krb5kdc[24079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984681, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:21.506994 24092 init.cc:377] Logged in from keytab as kudu/127.1.210.126@KRBTEST.COM (short username kudu)
I20260806 02:51:21.508589 24092 webserver.cc:460] Webserver started at http://127.1.210.126:35537/ using document root <none> and password file <none>
I20260806 02:51:21.509315 24092 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:21.509500 24092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:21.509831 24092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:21.513233 24092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "983f488f55cb48ad99c5723fcc2b3d1c"
format_stamp: "Formatted at 2026-08-06 02:51:21 on dist-test-slave-8tdl"
server_key: "722d03b91c17d94f416b682d33ff2207"
server_key_iv: "7a7f452cad590687a37625bf072af522"
server_key_version: "encryptionkey@0"
I20260806 02:51:21.514091 24092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "983f488f55cb48ad99c5723fcc2b3d1c"
format_stamp: "Formatted at 2026-08-06 02:51:21 on dist-test-slave-8tdl"
server_key: "722d03b91c17d94f416b682d33ff2207"
server_key_iv: "7a7f452cad590687a37625bf072af522"
server_key_version: "encryptionkey@0"
I20260806 02:51:21.519829 24092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260806 02:51:21.523653 24107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:21.525277 24092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:51:21.525511 24092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "983f488f55cb48ad99c5723fcc2b3d1c"
format_stamp: "Formatted at 2026-08-06 02:51:21 on dist-test-slave-8tdl"
server_key: "722d03b91c17d94f416b682d33ff2207"
server_key_iv: "7a7f452cad590687a37625bf072af522"
server_key_version: "encryptionkey@0"
I20260806 02:51:21.525774 24092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:21.532435 24092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:21.537325 24092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:21.537640 24092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:21.558223 24092 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:42751
I20260806 02:51:21.558224 24159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:42751 every 8 connection(s)
I20260806 02:51:21.559863 24092 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:51:21.564038 24160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:21.567672 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 24092
I20260806 02:51:21.567866 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:51:21.568404 1865 external_mini_cluster.cc:1428] Setting key 58072993363df3656b41420719d5082d
I20260806 02:51:21.578159 24163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:21.570387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50817 (local address 127.1.210.126:42751)
0806 02:51:21.571239 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:21.571256 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:21.571311 (+ 55us) server_negotiation.cc:407] Connection header received
0806 02:51:21.573753 (+ 2442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:21.573792 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:21.574352 (+ 560us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:21.574895 (+ 543us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:21.575993 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:21.576006 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:21.576028 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:51:21.576058 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:21.576144 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:51:21.576174 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:21.576428 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:21.576568 (+ 140us) server_negotiation.cc:299] Negotiation successful
0806 02:51:21.576782 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":344,"threads_started":1}
I20260806 02:51:21.579048 24160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 983f488f55cb48ad99c5723fcc2b3d1c: Bootstrap starting.
I20260806 02:51:21.582376 1865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:51:21.582865 24160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 983f488f55cb48ad99c5723fcc2b3d1c: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:21.583946 24160 log.cc:824] T 00000000000000000000000000000000 P 983f488f55cb48ad99c5723fcc2b3d1c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:21.587495 24160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 983f488f55cb48ad99c5723fcc2b3d1c: No bootstrap required, opened a new log
I20260806 02:51:21.590025 24163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:21.587291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43497 (local address 127.1.210.126:42751)
0806 02:51:21.587486 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:21.587493 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:21.587838 (+ 345us) server_negotiation.cc:407] Connection header received
0806 02:51:21.588006 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:21.588018 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:21.588108 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:21.588231 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:21.589335 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:21.589341 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:21.589348 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:51:21.589373 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:21.589429 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:51:21.589450 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:21.589591 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:21.589651 (+ 60us) server_negotiation.cc:299] Negotiation successful
0806 02:51:21.589716 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 02:51:21.597057 24160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 983f488f55cb48ad99c5723fcc2b3d1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "983f488f55cb48ad99c5723fcc2b3d1c" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 42751 } }
I20260806 02:51:21.597491 24160 raft_consensus.cc:374] T 00000000000000000000000000000000 P 983f488f55cb48ad99c5723fcc2b3d1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:21.597620 24160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 983f488f55cb48ad99c5723fcc2b3d1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 983f488f55cb48ad99c5723fcc2b3d1c, State: Initialized, Role: FOLLOWER
I20260806 02:51:21.598272 24160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 983f488f55cb48ad99c5723fcc2b3d1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "983f488f55cb48ad99c5723fcc2b3d1c" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 42751 } }
I20260806 02:51:21.598531 24160 raft_consensus.cc:388] T 00000000000000000000000000000000 P 983f488f55cb48ad99c5723fcc2b3d1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:51:21.598719 24160 raft_consensus.cc:482] T 00000000000000000000000000000000 P 983f488f55cb48ad99c5723fcc2b3d1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:51:21.598889 24160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 983f488f55cb48ad99c5723fcc2b3d1c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:21.601699 24160 raft_consensus.cc:504] T 00000000000000000000000000000000 P 983f488f55cb48ad99c5723fcc2b3d1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "983f488f55cb48ad99c5723fcc2b3d1c" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 42751 } }
I20260806 02:51:21.602384 24160 leader_election.cc:302] T 00000000000000000000000000000000 P 983f488f55cb48ad99c5723fcc2b3d1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 983f488f55cb48ad99c5723fcc2b3d1c; no voters:
I20260806 02:51:21.603122 24160 leader_election.cc:288] T 00000000000000000000000000000000 P 983f488f55cb48ad99c5723fcc2b3d1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:51:21.603277 24170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 983f488f55cb48ad99c5723fcc2b3d1c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:51:21.604001 24170 raft_consensus.cc:686] T 00000000000000000000000000000000 P 983f488f55cb48ad99c5723fcc2b3d1c [term 1 LEADER]: Becoming Leader. State: Replica: 983f488f55cb48ad99c5723fcc2b3d1c, State: Running, Role: LEADER
I20260806 02:51:21.604676 24170 consensus_queue.cc:237] T 00000000000000000000000000000000 P 983f488f55cb48ad99c5723fcc2b3d1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "983f488f55cb48ad99c5723fcc2b3d1c" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 42751 } }
I20260806 02:51:21.605360 24160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 983f488f55cb48ad99c5723fcc2b3d1c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:51:21.607688 24171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 983f488f55cb48ad99c5723fcc2b3d1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "983f488f55cb48ad99c5723fcc2b3d1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "983f488f55cb48ad99c5723fcc2b3d1c" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 42751 } } }
I20260806 02:51:21.607800 24172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 983f488f55cb48ad99c5723fcc2b3d1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 983f488f55cb48ad99c5723fcc2b3d1c. Latest consensus state: current_term: 1 leader_uuid: "983f488f55cb48ad99c5723fcc2b3d1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "983f488f55cb48ad99c5723fcc2b3d1c" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 42751 } } }
I20260806 02:51:21.608240 24172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 983f488f55cb48ad99c5723fcc2b3d1c [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:21.608218 24171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 983f488f55cb48ad99c5723fcc2b3d1c [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:21.609995 24175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:51:21.615913 24175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:51:21.625913 24175 catalog_manager.cc:1347] Generated new cluster ID: c4d45b10955144fcbf9def75e404b344
I20260806 02:51:21.626050 24175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:51:21.662810 24175 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:51:21.663404 24175 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:51:21.672989 24175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 983f488f55cb48ad99c5723fcc2b3d1c: Generated new TSK 0
I20260806 02:51:21.673923 24175 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:51:21.680137 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 24092
W20260806 02:51:21.681367 24155 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50817
W20260806 02:51:21.682184 1865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 24092: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:51:21.682379 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 24092
2026-08-06T02:51:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2073 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:51:21 dist-test-slave-8tdl krb5kdc[24192](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:51:21 dist-test-slave-8tdl krb5kdc[24192](info): set up 2 sockets
Aug 06 02:51:21 dist-test-slave-8tdl krb5kdc[24192](info): commencing operation
krb5kdc: starting...
W20260806 02:51:22.838466 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.006s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:51:23 dist-test-slave-8tdl krb5kdc[24192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:51:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:51:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
I20260806 02:51:23.183844 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:38497
--webserver_interface=127.1.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:38457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:38497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:23.549734 24205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:23.550120 24205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:23.550289 24205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:23.558394 24205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:51:23.558599 24205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:23.558741 24205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:23.558861 24205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:51:23.558951 24205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:51:23.569660 24205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:38457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:38497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:38497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:51:23.571264 24205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:23.572708 24205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:23.585367 24210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:23.585282 24213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:23.585335 24211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:23.586086 24205 server_base.cc:1034] running on GCE node
I20260806 02:51:23.587002 24205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:23.588508 24205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:23.589733 24205 hybrid_clock.cc:648] HybridClock initialized: now 1785984683589671 us; error 63 us; skew 500 ppm
I20260806 02:51:23.590078 24205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:51:23 dist-test-slave-8tdl krb5kdc[24192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984683, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:23.597529 24205 init.cc:377] Logged in from keytab as kudu/127.1.210.126@KRBTEST.COM (short username kudu)
I20260806 02:51:23.599354 24205 webserver.cc:460] Webserver started at http://127.1.210.126:44101/ using document root <none> and password file <none>
I20260806 02:51:23.600111 24205 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:23.600257 24205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:23.600703 24205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:23.604089 24205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "91934c826b3b4e879f38c10f9f142fa9"
format_stamp: "Formatted at 2026-08-06 02:51:23 on dist-test-slave-8tdl"
server_key: "bd91e90ea6dc891e8129be2def6c4123"
server_key_iv: "e258eb12f25f9b411672d87bf5458a35"
server_key_version: "encryptionkey@0"
I20260806 02:51:23.604943 24205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "91934c826b3b4e879f38c10f9f142fa9"
format_stamp: "Formatted at 2026-08-06 02:51:23 on dist-test-slave-8tdl"
server_key: "bd91e90ea6dc891e8129be2def6c4123"
server_key_iv: "e258eb12f25f9b411672d87bf5458a35"
server_key_version: "encryptionkey@0"
I20260806 02:51:23.610603 24205 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 02:51:23.614475 24220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:23.615993 24205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:51:23.616191 24205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "91934c826b3b4e879f38c10f9f142fa9"
format_stamp: "Formatted at 2026-08-06 02:51:23 on dist-test-slave-8tdl"
server_key: "bd91e90ea6dc891e8129be2def6c4123"
server_key_iv: "e258eb12f25f9b411672d87bf5458a35"
server_key_version: "encryptionkey@0"
I20260806 02:51:23.616418 24205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:23.645442 24205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:23.650324 24205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:23.650723 24205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:23.671577 24205 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:38497
I20260806 02:51:23.671612 24271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:38497 every 8 connection(s)
I20260806 02:51:23.673112 24205 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:51:23.673547 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 24205
I20260806 02:51:23.673725 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:51:23.674256 1865 external_mini_cluster.cc:1428] Setting key 97bbc3248cf6a334ab039407c5466b09
I20260806 02:51:23.677933 24272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 02:51:23 dist-test-slave-8tdl krb5kdc[24192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:51:23.689085 24272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91934c826b3b4e879f38c10f9f142fa9: Bootstrap starting.
I20260806 02:51:23.693571 24272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91934c826b3b4e879f38c10f9f142fa9: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:23.694928 24272 log.cc:824] T 00000000000000000000000000000000 P 91934c826b3b4e879f38c10f9f142fa9: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:23.698536 24272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91934c826b3b4e879f38c10f9f142fa9: No bootstrap required, opened a new log
I20260806 02:51:23.698776 24275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:23.676373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47709 (local address 127.1.210.126:38497)
0806 02:51:23.677185 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:23.677197 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:23.677239 (+ 42us) server_negotiation.cc:407] Connection header received
0806 02:51:23.679089 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:23.679112 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:23.679511 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:23.679919 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:23.681527 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:23.682968 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:23.684288 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:23.684992 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:23.685188 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:23.689542 (+ 4354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:23.689555 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:23.689579 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:23.689635 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:23.693886 (+ 4251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:23.694610 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:23.694630 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:23.694642 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:23.694944 (+ 302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:23.695431 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:23.695438 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:23.695444 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:23.696098 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:23.696404 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:23.696925 (+ 521us) server_negotiation.cc:299] Negotiation successful
0806 02:51:23.697392 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":360,"threads_started":1}
I20260806 02:51:23.702834 1865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:51:23.707347 24272 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91934c826b3b4e879f38c10f9f142fa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91934c826b3b4e879f38c10f9f142fa9" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 38497 } }
I20260806 02:51:23.707897 24272 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91934c826b3b4e879f38c10f9f142fa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:23.708026 24272 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91934c826b3b4e879f38c10f9f142fa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91934c826b3b4e879f38c10f9f142fa9, State: Initialized, Role: FOLLOWER
I20260806 02:51:23.708727 24272 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91934c826b3b4e879f38c10f9f142fa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91934c826b3b4e879f38c10f9f142fa9" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 38497 } }
I20260806 02:51:23.709003 24272 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91934c826b3b4e879f38c10f9f142fa9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:51:23.709161 24272 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91934c826b3b4e879f38c10f9f142fa9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:51:23.709336 24272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91934c826b3b4e879f38c10f9f142fa9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:23.711819 24272 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91934c826b3b4e879f38c10f9f142fa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91934c826b3b4e879f38c10f9f142fa9" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 38497 } }
I20260806 02:51:23.712352 24272 leader_election.cc:302] T 00000000000000000000000000000000 P 91934c826b3b4e879f38c10f9f142fa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91934c826b3b4e879f38c10f9f142fa9; no voters:
I20260806 02:51:23.713078 24272 leader_election.cc:288] T 00000000000000000000000000000000 P 91934c826b3b4e879f38c10f9f142fa9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:51:23.713212 24282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91934c826b3b4e879f38c10f9f142fa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:51:23.714994 24282 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91934c826b3b4e879f38c10f9f142fa9 [term 1 LEADER]: Becoming Leader. State: Replica: 91934c826b3b4e879f38c10f9f142fa9, State: Running, Role: LEADER
I20260806 02:51:23.715790 24272 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91934c826b3b4e879f38c10f9f142fa9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:51:23.715644 24282 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91934c826b3b4e879f38c10f9f142fa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91934c826b3b4e879f38c10f9f142fa9" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 38497 } }
I20260806 02:51:23.720400 24283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91934c826b3b4e879f38c10f9f142fa9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91934c826b3b4e879f38c10f9f142fa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91934c826b3b4e879f38c10f9f142fa9" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 38497 } } }
I20260806 02:51:23.723639 24283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91934c826b3b4e879f38c10f9f142fa9 [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:23.727541 24275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:23.707402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43705 (local address 127.1.210.126:38497)
0806 02:51:23.707635 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:23.707646 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:23.708082 (+ 436us) server_negotiation.cc:407] Connection header received
0806 02:51:23.708286 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:23.708291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:23.708359 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:23.708480 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:23.710224 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:23.711113 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:23.712491 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:23.713082 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:23.713229 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:23.716090 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:23.716095 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:23.716099 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:23.716130 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:23.719777 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:23.720718 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:23.720723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:23.720726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:23.723140 (+ 2414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:23.723696 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:23.723701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:23.723706 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:23.724150 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:23.726521 (+ 2371us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:23.726727 (+ 206us) server_negotiation.cc:299] Negotiation successful
0806 02:51:23.727049 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 02:51:23.728027 24290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:51:23.728994 24284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91934c826b3b4e879f38c10f9f142fa9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91934c826b3b4e879f38c10f9f142fa9. Latest consensus state: current_term: 1 leader_uuid: "91934c826b3b4e879f38c10f9f142fa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91934c826b3b4e879f38c10f9f142fa9" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 38497 } } }
I20260806 02:51:23.729372 24284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91934c826b3b4e879f38c10f9f142fa9 [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:23.736289 24290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:51:23.747290 24290 catalog_manager.cc:1347] Generated new cluster ID: 3ee29b69f44a4981bb5948e2869153c2
I20260806 02:51:23.747450 24290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:51:23.764322 24290 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:51:23.765777 24290 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:51:23.773878 24290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91934c826b3b4e879f38c10f9f142fa9: Generated new TSK 0
I20260806 02:51:23.774842 24290 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:51:23.785259 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 24205
I20260806 02:51:23.786437 24267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:51:23.881260 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 24205
2026-08-06T02:51:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2202 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:51:24 dist-test-slave-8tdl krb5kdc[24304](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:51:24 dist-test-slave-8tdl krb5kdc[24304](info): set up 2 sockets
Aug 06 02:51:24 dist-test-slave-8tdl krb5kdc[24304](info): commencing operation
krb5kdc: starting...
W20260806 02:51:25.054472 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:51:25 dist-test-slave-8tdl krb5kdc[24304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:51:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:51:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
I20260806 02:51:25.390642 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:45731
--webserver_interface=127.1.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:43357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:45731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:25.750386 24317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:25.750710 24317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:25.750875 24317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:25.758316 24317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:51:25.758445 24317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:25.758530 24317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:25.758625 24317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:51:25.758709 24317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:51:25.769137 24317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:43357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:45731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:45731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:51:25.770572 24317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:25.772047 24317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:25.784821 24322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:25.784798 24326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:25.784729 24324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:25.785835 24317 server_base.cc:1034] running on GCE node
I20260806 02:51:25.786767 24317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:25.788300 24317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:25.789577 24317 hybrid_clock.cc:648] HybridClock initialized: now 1785984685789517 us; error 47 us; skew 500 ppm
I20260806 02:51:25.790021 24317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:51:25 dist-test-slave-8tdl krb5kdc[24304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984685, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:25.793663 24317 init.cc:377] Logged in from keytab as kudu/127.1.210.126@KRBTEST.COM (short username kudu)
I20260806 02:51:25.795457 24317 webserver.cc:460] Webserver started at http://127.1.210.126:46279/ using document root <none> and password file <none>
I20260806 02:51:25.796214 24317 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:25.796373 24317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:25.796715 24317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:25.799983 24317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "08720ac2f927466cb514c24a93d04d40"
format_stamp: "Formatted at 2026-08-06 02:51:25 on dist-test-slave-8tdl"
server_key: "27a1e913b61e5d4dd2598a34ba5efc99"
server_key_iv: "2c6f91c8531ac8e09f39586ff84f7783"
server_key_version: "encryptionkey@0"
I20260806 02:51:25.800858 24317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "08720ac2f927466cb514c24a93d04d40"
format_stamp: "Formatted at 2026-08-06 02:51:25 on dist-test-slave-8tdl"
server_key: "27a1e913b61e5d4dd2598a34ba5efc99"
server_key_iv: "2c6f91c8531ac8e09f39586ff84f7783"
server_key_version: "encryptionkey@0"
I20260806 02:51:25.806900 24317 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 02:51:25.810583 24333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:25.812140 24317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:51:25.812341 24317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "08720ac2f927466cb514c24a93d04d40"
format_stamp: "Formatted at 2026-08-06 02:51:25 on dist-test-slave-8tdl"
server_key: "27a1e913b61e5d4dd2598a34ba5efc99"
server_key_iv: "2c6f91c8531ac8e09f39586ff84f7783"
server_key_version: "encryptionkey@0"
I20260806 02:51:25.812597 24317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:25.828866 24317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:25.833914 24317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:25.834251 24317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:25.855201 24317 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:45731
I20260806 02:51:25.855232 24385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:45731 every 8 connection(s)
I20260806 02:51:25.856678 24317 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:51:25.861019 24386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:25.863865 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 24317
I20260806 02:51:25.864071 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:51:25.864528 1865 external_mini_cluster.cc:1428] Setting key 0d8bc3399c347767f873a01e9074d6b3
I20260806 02:51:25.876458 24386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08720ac2f927466cb514c24a93d04d40: Bootstrap starting.
I20260806 02:51:25.878290 24389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:25.866493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56023 (local address 127.1.210.126:45731)
0806 02:51:25.867468 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:25.867480 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:25.867527 (+ 47us) server_negotiation.cc:407] Connection header received
0806 02:51:25.869305 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:25.869329 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:25.869779 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:25.870228 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:25.870810 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:25.872137 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:25.873517 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:25.874273 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:25.874471 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:25.875890 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:25.875898 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:25.875913 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:51:25.876525 (+ 612us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:25.876622 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:51:25.876661 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:25.876782 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:25.877023 (+ 241us) server_negotiation.cc:299] Negotiation successful
0806 02:51:25.877317 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":379,"threads_started":1}
I20260806 02:51:25.880131 24386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08720ac2f927466cb514c24a93d04d40: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:25.881640 24386 log.cc:824] T 00000000000000000000000000000000 P 08720ac2f927466cb514c24a93d04d40: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:25.882669 1865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:51:25.885453 24386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08720ac2f927466cb514c24a93d04d40: No bootstrap required, opened a new log
I20260806 02:51:25.896467 24389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:25.888835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52613 (local address 127.1.210.126:45731)
0806 02:51:25.889090 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:25.889102 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:25.889766 (+ 664us) server_negotiation.cc:407] Connection header received
0806 02:51:25.889970 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:25.889975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:25.890041 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:25.890135 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:25.890844 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:25.891635 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:25.893797 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:25.894283 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:25.894380 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:25.895631 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:25.895636 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:25.895640 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:51:25.895660 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:25.895699 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:51:25.895714 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:25.895790 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:25.895992 (+ 202us) server_negotiation.cc:299] Negotiation successful
0806 02:51:25.896070 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 02:51:25.897889 24386 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08720ac2f927466cb514c24a93d04d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08720ac2f927466cb514c24a93d04d40" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45731 } }
I20260806 02:51:25.898399 24386 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08720ac2f927466cb514c24a93d04d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:25.898522 24386 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08720ac2f927466cb514c24a93d04d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08720ac2f927466cb514c24a93d04d40, State: Initialized, Role: FOLLOWER
I20260806 02:51:25.899237 24386 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08720ac2f927466cb514c24a93d04d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08720ac2f927466cb514c24a93d04d40" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45731 } }
I20260806 02:51:25.899507 24386 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08720ac2f927466cb514c24a93d04d40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:51:25.899683 24386 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08720ac2f927466cb514c24a93d04d40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:51:25.899883 24386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08720ac2f927466cb514c24a93d04d40 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:25.902580 24386 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08720ac2f927466cb514c24a93d04d40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08720ac2f927466cb514c24a93d04d40" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45731 } }
I20260806 02:51:25.903178 24386 leader_election.cc:302] T 00000000000000000000000000000000 P 08720ac2f927466cb514c24a93d04d40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08720ac2f927466cb514c24a93d04d40; no voters:
I20260806 02:51:25.903795 24386 leader_election.cc:288] T 00000000000000000000000000000000 P 08720ac2f927466cb514c24a93d04d40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:51:25.903964 24396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08720ac2f927466cb514c24a93d04d40 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:51:25.905333 24396 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08720ac2f927466cb514c24a93d04d40 [term 1 LEADER]: Becoming Leader. State: Replica: 08720ac2f927466cb514c24a93d04d40, State: Running, Role: LEADER
I20260806 02:51:25.905906 24396 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08720ac2f927466cb514c24a93d04d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08720ac2f927466cb514c24a93d04d40" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45731 } }
I20260806 02:51:25.906518 24386 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08720ac2f927466cb514c24a93d04d40 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:51:25.909222 24397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08720ac2f927466cb514c24a93d04d40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08720ac2f927466cb514c24a93d04d40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08720ac2f927466cb514c24a93d04d40" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45731 } } }
I20260806 02:51:25.909260 24398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08720ac2f927466cb514c24a93d04d40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08720ac2f927466cb514c24a93d04d40. Latest consensus state: current_term: 1 leader_uuid: "08720ac2f927466cb514c24a93d04d40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08720ac2f927466cb514c24a93d04d40" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 45731 } } }
I20260806 02:51:25.909675 24398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08720ac2f927466cb514c24a93d04d40 [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:25.909675 24397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08720ac2f927466cb514c24a93d04d40 [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:25.910727 24404 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:51:25.921124 24404 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:51:25.931212 24404 catalog_manager.cc:1347] Generated new cluster ID: f7305785ebe849a0b83e8165db4a361c
I20260806 02:51:25.931365 24404 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:51:25.992008 24404 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:51:25.994040 24404 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:51:26.002199 24404 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08720ac2f927466cb514c24a93d04d40: Generated new TSK 0
I20260806 02:51:26.003425 24404 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:51:26.019811 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 24317
W20260806 02:51:26.021780 24381 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56023
W20260806 02:51:26.022639 1865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 24317: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:51:26.022833 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 24317
2026-08-06T02:51:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2141 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:51:26 dist-test-slave-8tdl krb5kdc[24417](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:51:26 dist-test-slave-8tdl krb5kdc[24417](info): set up 2 sockets
Aug 06 02:51:26 dist-test-slave-8tdl krb5kdc[24417](info): commencing operation
krb5kdc: starting...
W20260806 02:51:27.170352 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.005s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:51:27 dist-test-slave-8tdl krb5kdc[24417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:51:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:51:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
I20260806 02:51:27.511461 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:40923
--webserver_interface=127.1.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:32911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:40923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:27.874274 24430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:27.874595 24430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:27.874794 24430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:27.882344 24430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:51:27.882478 24430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:27.882614 24430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:27.882756 24430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:51:27.882820 24430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:51:27.893203 24430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:32911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:40923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:40923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:51:27.894590 24430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:27.896049 24430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:27.908655 24436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:27.908593 24435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:27.908913 24438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:27.909962 24430 server_base.cc:1034] running on GCE node
I20260806 02:51:27.910948 24430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:27.912736 24430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:27.914113 24430 hybrid_clock.cc:648] HybridClock initialized: now 1785984687914048 us; error 60 us; skew 500 ppm
I20260806 02:51:27.914513 24430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:51:27 dist-test-slave-8tdl krb5kdc[24417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984687, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:27.917954 24430 init.cc:377] Logged in from keytab as kudu/127.1.210.126@KRBTEST.COM (short username kudu)
I20260806 02:51:27.919651 24430 webserver.cc:460] Webserver started at http://127.1.210.126:40695/ using document root <none> and password file <none>
I20260806 02:51:27.920315 24430 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:27.920421 24430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:27.920688 24430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:27.924027 24430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "8a60178c684c4b999f18e3ae4851670a"
format_stamp: "Formatted at 2026-08-06 02:51:27 on dist-test-slave-8tdl"
server_key: "d821c79e913fe0a4b46064100ab764ee"
server_key_iv: "c94c0169166af110ddc5e71274984c99"
server_key_version: "encryptionkey@0"
I20260806 02:51:27.924803 24430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "8a60178c684c4b999f18e3ae4851670a"
format_stamp: "Formatted at 2026-08-06 02:51:27 on dist-test-slave-8tdl"
server_key: "d821c79e913fe0a4b46064100ab764ee"
server_key_iv: "c94c0169166af110ddc5e71274984c99"
server_key_version: "encryptionkey@0"
I20260806 02:51:27.930464 24430 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 02:51:27.934566 24445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:27.936146 24430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:51:27.936345 24430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "8a60178c684c4b999f18e3ae4851670a"
format_stamp: "Formatted at 2026-08-06 02:51:27 on dist-test-slave-8tdl"
server_key: "d821c79e913fe0a4b46064100ab764ee"
server_key_iv: "c94c0169166af110ddc5e71274984c99"
server_key_version: "encryptionkey@0"
I20260806 02:51:27.936587 24430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:27.943035 24430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:27.947929 24430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:27.948252 24430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:27.968377 24430 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:40923
I20260806 02:51:27.968401 24497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:40923 every 8 connection(s)
I20260806 02:51:27.969859 24430 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:51:27.973794 24498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:27.974406 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 24430
I20260806 02:51:27.974601 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:51:27.975199 1865 external_mini_cluster.cc:1428] Setting key f20bedb4bb15ca8e9e4a4e3a209d4ec4
I20260806 02:51:27.983834 24501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:27.977115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36799 (local address 127.1.210.126:40923)
0806 02:51:27.977964 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:27.977980 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:27.978032 (+ 52us) server_negotiation.cc:407] Connection header received
0806 02:51:27.979807 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:27.979844 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:27.980339 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:27.980822 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:27.981858 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:27.981869 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:27.981891 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:51:27.981909 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:27.981980 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:51:27.982006 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:27.982264 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:27.982412 (+ 148us) server_negotiation.cc:299] Negotiation successful
0806 02:51:27.982610 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":333,"threads_started":1}
I20260806 02:51:27.987584 1865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:51:27.989259 24498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a60178c684c4b999f18e3ae4851670a: Bootstrap starting.
I20260806 02:51:27.992741 24498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a60178c684c4b999f18e3ae4851670a: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:27.994048 24498 log.cc:824] T 00000000000000000000000000000000 P 8a60178c684c4b999f18e3ae4851670a: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:27.995389 24501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:27.992144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45231 (local address 127.1.210.126:40923)
0806 02:51:27.992326 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:27.992332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:27.992904 (+ 572us) server_negotiation.cc:407] Connection header received
0806 02:51:27.993099 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:27.993106 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:27.993189 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:27.993319 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:27.994498 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:27.994505 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:27.994510 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:51:27.994533 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:27.994599 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:51:27.994629 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:27.994776 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:27.994969 (+ 193us) server_negotiation.cc:299] Negotiation successful
0806 02:51:27.995015 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 02:51:27.997561 24498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a60178c684c4b999f18e3ae4851670a: No bootstrap required, opened a new log
I20260806 02:51:28.005285 24498 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a60178c684c4b999f18e3ae4851670a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a60178c684c4b999f18e3ae4851670a" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 40923 } }
I20260806 02:51:28.005699 24498 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a60178c684c4b999f18e3ae4851670a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:28.005852 24498 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a60178c684c4b999f18e3ae4851670a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a60178c684c4b999f18e3ae4851670a, State: Initialized, Role: FOLLOWER
I20260806 02:51:28.006601 24498 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a60178c684c4b999f18e3ae4851670a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a60178c684c4b999f18e3ae4851670a" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 40923 } }
I20260806 02:51:28.006975 24498 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a60178c684c4b999f18e3ae4851670a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:51:28.007154 24498 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a60178c684c4b999f18e3ae4851670a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:51:28.007337 24498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a60178c684c4b999f18e3ae4851670a [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:28.009676 24498 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a60178c684c4b999f18e3ae4851670a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a60178c684c4b999f18e3ae4851670a" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 40923 } }
I20260806 02:51:28.010237 24498 leader_election.cc:302] T 00000000000000000000000000000000 P 8a60178c684c4b999f18e3ae4851670a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a60178c684c4b999f18e3ae4851670a; no voters:
I20260806 02:51:28.010998 24498 leader_election.cc:288] T 00000000000000000000000000000000 P 8a60178c684c4b999f18e3ae4851670a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:51:28.011168 24508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a60178c684c4b999f18e3ae4851670a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:51:28.011932 24508 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a60178c684c4b999f18e3ae4851670a [term 1 LEADER]: Becoming Leader. State: Replica: 8a60178c684c4b999f18e3ae4851670a, State: Running, Role: LEADER
I20260806 02:51:28.012567 24508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a60178c684c4b999f18e3ae4851670a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a60178c684c4b999f18e3ae4851670a" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 40923 } }
I20260806 02:51:28.013429 24498 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a60178c684c4b999f18e3ae4851670a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:51:28.017102 24510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a60178c684c4b999f18e3ae4851670a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a60178c684c4b999f18e3ae4851670a. Latest consensus state: current_term: 1 leader_uuid: "8a60178c684c4b999f18e3ae4851670a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a60178c684c4b999f18e3ae4851670a" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 40923 } } }
I20260806 02:51:28.017201 24509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a60178c684c4b999f18e3ae4851670a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a60178c684c4b999f18e3ae4851670a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a60178c684c4b999f18e3ae4851670a" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 40923 } } }
I20260806 02:51:28.017567 24510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a60178c684c4b999f18e3ae4851670a [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:28.017639 24509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a60178c684c4b999f18e3ae4851670a [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:28.019801 24515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:51:28.025663 24515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:51:28.035543 24515 catalog_manager.cc:1347] Generated new cluster ID: 58be8128700840acad16f3b91fcdad8f
I20260806 02:51:28.035689 24515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:51:28.055796 24515 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:51:28.056321 24515 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:51:28.065132 24515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a60178c684c4b999f18e3ae4851670a: Generated new TSK 0
I20260806 02:51:28.066061 24515 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:51:28.080955 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 24430
W20260806 02:51:28.082458 24493 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36799
W20260806 02:51:28.083350 1865 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 24430: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:51:28.083554 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 24430
2026-08-06T02:51:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2060 ms)
[----------] 18 tests from AuthTokenIssuingTest (38414 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:51:28 dist-test-slave-8tdl krb5kdc[24529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:51:28 dist-test-slave-8tdl krb5kdc[24529](info): set up 2 sockets
Aug 06 02:51:28 dist-test-slave-8tdl krb5kdc[24529](info): commencing operation
krb5kdc: starting...
W20260806 02:51:29.250773 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.008s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:51:29 dist-test-slave-8tdl krb5kdc[24529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:51:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:51:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
I20260806 02:51:29.595530 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:37905
--webserver_interface=127.1.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:43447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:37905,127.1.210.125:41551,127.1.210.124:35079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:29.959769 24542 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 02:51:29.960177 24542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:29.960309 24542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:29.960448 24542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:29.968130 24542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:51:29.968271 24542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:29.968393 24542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:29.968485 24542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:51:29.968595 24542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:51:29.979753 24542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:43447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:37905,127.1.210.125:41551,127.1.210.124:35079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:37905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:51:29.981159 24542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:29.982543 24542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:29.994658 24547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:29.994779 24550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:29.994830 24548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:29.996194 24542 server_base.cc:1034] running on GCE node
I20260806 02:51:29.997069 24542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:29.998589 24542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:29.999856 24542 hybrid_clock.cc:648] HybridClock initialized: now 1785984689999772 us; error 32 us; skew 500 ppm
I20260806 02:51:30.000303 24542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:51:30 dist-test-slave-8tdl krb5kdc[24529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984690, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:30.003882 24542 init.cc:377] Logged in from keytab as kudu/127.1.210.126@KRBTEST.COM (short username kudu)
I20260806 02:51:30.005683 24542 webserver.cc:460] Webserver started at http://127.1.210.126:33557/ using document root <none> and password file <none>
I20260806 02:51:30.006456 24542 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:30.006553 24542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:30.006953 24542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:30.010185 24542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "12a7d16c7b9742728d9faea61a7ad675"
format_stamp: "Formatted at 2026-08-06 02:51:30 on dist-test-slave-8tdl"
server_key: "17a214f41e91ec89828101f6f5345d52"
server_key_iv: "5d05f5c293897d14da6f227a720fd253"
server_key_version: "encryptionkey@0"
I20260806 02:51:30.011128 24542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "12a7d16c7b9742728d9faea61a7ad675"
format_stamp: "Formatted at 2026-08-06 02:51:30 on dist-test-slave-8tdl"
server_key: "17a214f41e91ec89828101f6f5345d52"
server_key_iv: "5d05f5c293897d14da6f227a720fd253"
server_key_version: "encryptionkey@0"
I20260806 02:51:30.017246 24542 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 02:51:30.021091 24557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:30.022763 24542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:51:30.022964 24542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "12a7d16c7b9742728d9faea61a7ad675"
format_stamp: "Formatted at 2026-08-06 02:51:30 on dist-test-slave-8tdl"
server_key: "17a214f41e91ec89828101f6f5345d52"
server_key_iv: "5d05f5c293897d14da6f227a720fd253"
server_key_version: "encryptionkey@0"
I20260806 02:51:30.023259 24542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:30.045329 24542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:30.050239 24542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:30.050540 24542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:30.072276 24542 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:37905
I20260806 02:51:30.072288 24609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:37905 every 8 connection(s)
I20260806 02:51:30.073789 24542 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:51:30.077916 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:30.080504 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 24542
I20260806 02:51:30.080751 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:51:30.081439 1865 external_mini_cluster.cc:1428] Setting key 3d883ede34bbc6a3a8ab2bdcdf1e7778
I20260806 02:51:30.085438 24610 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.126" port: 37905 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:51:30 dist-test-slave-8tdl krb5kdc[24529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984690, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:51:30.111248 24613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:30.090585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44766 (local address 127.1.210.126:37905)
0806 02:51:30.091347 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:30.091357 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:30.091422 (+ 65us) server_negotiation.cc:407] Connection header received
0806 02:51:30.092877 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:30.092883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:30.093144 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:30.093363 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:30.094760 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:30.096001 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:30.097426 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:30.098003 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:30.098214 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:30.102482 (+ 4268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:30.102502 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:30.102517 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:30.102556 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:30.105986 (+ 3430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:30.107269 (+ 1283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:30.107276 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:30.107278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:30.107462 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:30.108020 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:30.108024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:30.108026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:30.108653 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:30.108836 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:30.109667 (+ 831us) server_negotiation.cc:299] Negotiation successful
0806 02:51:30.109985 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":536,"thread_start_us":332,"threads_started":1}
I20260806 02:51:30.111286 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:30.089133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:37905 (local address 127.0.0.1:44766)
0806 02:51:30.090066 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:30.090111 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:30.092461 (+ 2350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:30.093375 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:30.093382 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:30.094135 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:30.094499 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:30.094518 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:30.096173 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:30.096183 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:30.097276 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:30.097289 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:30.098246 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:30.098252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:30.098469 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:30.099219 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:30.099243 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:30.102069 (+ 2826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:30.106275 (+ 4206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:30.106292 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:30.106310 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:30.107099 (+ 789us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:30.107599 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:30.107606 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:30.107612 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:30.107855 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:30.108820 (+ 965us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:30.108831 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:30.109126 (+ 295us) client_negotiation.cc:769] Sending connection context
0806 02:51:30.109462 (+ 336us) client_negotiation.cc:240] Negotiation successful
0806 02:51:30.110201 (+ 739us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":129,"thread_start_us":396,"threads_started":1}
WARNING: no policy specified for kudu/127.1.210.125@KRBTEST.COM; defaulting to no policy
I20260806 02:51:30.115629 24610 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.125" port: 41551 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:51:30.117146 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:30.116652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:41551 (local address 127.0.0.1:39484)
0806 02:51:30.116855 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:30.116997 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.125:41551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.125@KRBTEST.COM" created.
W20260806 02:51:30.117859 24561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.210.125:41551: connect: Connection refused (error 111)
I20260806 02:51:30.119863 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:30.119296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:41551 (local address 127.0.0.1:39490)
0806 02:51:30.119535 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:30.119647 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.125:41551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260806 02:51:30.120183 24610 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.125:41551: Network error: Client connection negotiation failed: client connection to 127.1.210.125:41551: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.125.keytab.
Entry for principal kudu/127.1.210.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.125.keytab.
I20260806 02:51:30.159022 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.125:41551
--webserver_interface=127.1.210.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.125.keytab
--principal=kudu/127.1.210.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:43447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:37905,127.1.210.125:41551,127.1.210.124:35079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:51:30.169638 24610 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.125" port: 41551 } attempt: 1
I20260806 02:51:30.171046 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:30.170448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:41551 (local address 127.0.0.1:39492)
0806 02:51:30.170725 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:30.170853 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.125:41551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260806 02:51:30.172170 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:30.171766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:41551 (local address 127.0.0.1:39494)
0806 02:51:30.171920 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:30.172007 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.125:41551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 02:51:30.172425 24610 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.125:41551: Network error: Client connection negotiation failed: client connection to 127.1.210.125:41551: connect: Connection refused (error 111)
I20260806 02:51:30.240753 24610 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.125" port: 41551 } attempt: 2
I20260806 02:51:30.242261 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:30.241650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:41551 (local address 127.0.0.1:39498)
0806 02:51:30.241936 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:30.242068 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.125:41551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260806 02:51:30.243607 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:30.243107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:41551 (local address 127.0.0.1:39514)
0806 02:51:30.243296 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:30.243409 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.125:41551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260806 02:51:30.243880 24610 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.125:41551: Network error: Client connection negotiation failed: client connection to 127.1.210.125:41551: connect: Connection refused (error 111)
I20260806 02:51:30.335199 24610 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.125" port: 41551 } attempt: 3
I20260806 02:51:30.336656 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:30.336008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:41551 (local address 127.0.0.1:39530)
0806 02:51:30.336288 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:30.336427 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.125:41551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260806 02:51:30.338198 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:30.337640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:41551 (local address 127.0.0.1:39546)
0806 02:51:30.337849 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:30.337967 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.125:41551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260806 02:51:30.338573 24610 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.125:41551: Network error: Client connection negotiation failed: client connection to 127.1.210.125:41551: connect: Connection refused (error 111)
I20260806 02:51:30.481909 24610 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.125" port: 41551 } attempt: 4
I20260806 02:51:30.483364 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:30.482623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:41551 (local address 127.0.0.1:39552)
0806 02:51:30.483062 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:30.483179 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.125:41551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264}
I20260806 02:51:30.484676 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:30.484226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:41551 (local address 127.0.0.1:39562)
0806 02:51:30.484399 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:30.484505 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.125:41551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260806 02:51:30.484925 24610 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.125:41551: Network error: Client connection negotiation failed: client connection to 127.1.210.125:41551: connect: Connection refused (error 111)
W20260806 02:51:30.515865 24617 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 02:51:30.516203 24617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:30.516270 24617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:30.516355 24617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:30.523865 24617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:51:30.523958 24617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:30.524025 24617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:30.524080 24617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:51:30.524137 24617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:51:30.534379 24617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:43447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:37905,127.1.210.125:41551,127.1.210.124:35079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.125.keytab
--principal=kudu/127.1.210.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.125:41551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.210.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:51:30.535807 24617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:30.537089 24617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:30.549597 24623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:30.549679 24626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:30.549716 24624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:30.550829 24617 server_base.cc:1034] running on GCE node
I20260806 02:51:30.551699 24617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:30.553212 24617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:30.554493 24617 hybrid_clock.cc:648] HybridClock initialized: now 1785984690554419 us; error 69 us; skew 500 ppm
I20260806 02:51:30.554939 24617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:51:30 dist-test-slave-8tdl krb5kdc[24529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984690, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:30.558380 24617 init.cc:377] Logged in from keytab as kudu/127.1.210.125@KRBTEST.COM (short username kudu)
I20260806 02:51:30.560150 24617 webserver.cc:460] Webserver started at http://127.1.210.125:40561/ using document root <none> and password file <none>
I20260806 02:51:30.560901 24617 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:30.561051 24617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:30.561398 24617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:30.564714 24617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-1/data/instance:
uuid: "3261570467c847edb61b03507bf07ed8"
format_stamp: "Formatted at 2026-08-06 02:51:30 on dist-test-slave-8tdl"
server_key: "d686e91520a43ca87c08cf88cb0898ce"
server_key_iv: "960d9820749338405878b87a0a1b2af7"
server_key_version: "encryptionkey@0"
I20260806 02:51:30.565662 24617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-1/wal/instance:
uuid: "3261570467c847edb61b03507bf07ed8"
format_stamp: "Formatted at 2026-08-06 02:51:30 on dist-test-slave-8tdl"
server_key: "d686e91520a43ca87c08cf88cb0898ce"
server_key_iv: "960d9820749338405878b87a0a1b2af7"
server_key_version: "encryptionkey@0"
I20260806 02:51:30.571489 24617 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 02:51:30.575446 24633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:30.577165 24617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:51:30.577394 24617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-1/wal
uuid: "3261570467c847edb61b03507bf07ed8"
format_stamp: "Formatted at 2026-08-06 02:51:30 on dist-test-slave-8tdl"
server_key: "d686e91520a43ca87c08cf88cb0898ce"
server_key_iv: "960d9820749338405878b87a0a1b2af7"
server_key_version: "encryptionkey@0"
I20260806 02:51:30.577648 24617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:30.600270 24617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:30.605463 24617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:30.605785 24617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:30.628036 24617 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.125:41551
I20260806 02:51:30.628100 24684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.125:41551 every 8 connection(s)
I20260806 02:51:30.629601 24617 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-1/data/info.pb
I20260806 02:51:30.633639 24685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:30.634092 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 24617
I20260806 02:51:30.634298 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-1/wal/instance
I20260806 02:51:30.634910 1865 external_mini_cluster.cc:1428] Setting key fcacc33f0a8e16825622e5a2e122b2e4
I20260806 02:51:30.644132 24685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.126" port: 37905 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:51:30 dist-test-slave-8tdl krb5kdc[24529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984690, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.125@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.210.124@KRBTEST.COM; defaulting to no policy
I20260806 02:51:30.670564 24687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:30.647826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44776 (local address 127.1.210.126:37905)
0806 02:51:30.648428 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:30.648437 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:30.649545 (+ 1108us) server_negotiation.cc:407] Connection header received
0806 02:51:30.652444 (+ 2899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:30.652448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:30.652523 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:30.652631 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:30.655063 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:30.655938 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:30.658579 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:30.659153 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:30.659267 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:30.664102 (+ 4835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:30.664107 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:30.664111 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:30.664164 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:30.667151 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:30.667974 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:30.667979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:30.667981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:30.668121 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:30.668615 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:30.668620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:30.668622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:30.668918 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:30.669091 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:30.669868 (+ 777us) server_negotiation.cc:299] Negotiation successful
0806 02:51:30.670112 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":298,"threads_started":1}
I20260806 02:51:30.671648 24688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:30.648426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:37905 (local address 127.0.0.1:44776)
0806 02:51:30.649341 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:30.649391 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:30.652104 (+ 2713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:30.652917 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:30.652931 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:30.653838 (+ 907us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:30.654748 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:30.654780 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:30.656124 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:30.656132 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:30.658376 (+ 2244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:30.658398 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:30.659412 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:30.659422 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:30.659767 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:30.660657 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:30.660693 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:30.663805 (+ 3112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:30.667318 (+ 3513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:30.667329 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:30.667347 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:30.667817 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:30.668242 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:30.668248 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:30.668251 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:30.668481 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:30.669065 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:30.669074 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:30.669466 (+ 392us) client_negotiation.cc:769] Sending connection context
0806 02:51:30.669768 (+ 302us) client_negotiation.cc:240] Negotiation successful
0806 02:51:30.670346 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":376,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.124@KRBTEST.COM" created.
I20260806 02:51:30.673720 24685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.125" port: 41551 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:51:30 dist-test-slave-8tdl krb5kdc[24529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984690, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.125@KRBTEST.COM for kudu/127.1.210.125@KRBTEST.COM
I20260806 02:51:30.690652 24688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:30.674740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:41551 (local address 127.0.0.1:39564)
0806 02:51:30.674952 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:30.674969 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:30.675159 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:30.676062 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:30.676067 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:30.676605 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:30.676751 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:30.676760 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:30.678408 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:30.678417 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:30.679424 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:30.679437 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:30.680382 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:30.680387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:30.680653 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:30.681319 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:30.681338 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:30.684073 (+ 2735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:30.687839 (+ 3766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:30.687847 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:30.687851 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:30.688272 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:30.688704 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:30.688713 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:30.688717 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:30.688869 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:30.689533 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:30.689538 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:30.689701 (+ 163us) client_negotiation.cc:769] Sending connection context
0806 02:51:30.689849 (+ 148us) client_negotiation.cc:240] Negotiation successful
0806 02:51:30.690166 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":26}
I20260806 02:51:30.690764 24690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:30.674808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39564 (local address 127.1.210.125:41551)
0806 02:51:30.675582 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:30.675589 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:30.675620 (+ 31us) server_negotiation.cc:407] Connection header received
0806 02:51:30.675669 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:30.675675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:30.675877 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:30.676076 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:30.677047 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:30.678257 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:30.679586 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:30.680219 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:30.680369 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:30.684307 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:30.684320 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:30.684334 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:30.684367 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:30.687653 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:30.688441 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:30.688454 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:30.688456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:30.688584 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:30.688994 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:30.689006 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:30.689009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:30.689396 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:30.689629 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:30.690036 (+ 407us) server_negotiation.cc:299] Negotiation successful
0806 02:51:30.690276 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":11,"server-negotiator.queue_time_us":588,"thread_start_us":327,"threads_started":1}
I20260806 02:51:30.693372 24685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.124" port: 35079 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:51:30.694819 24688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:30.694245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:35079 (local address 127.0.0.1:41818)
0806 02:51:30.694482 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:30.694590 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:35079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260806 02:51:30.695187 24636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.210.124:35079: connect: Connection refused (error 111)
I20260806 02:51:30.696936 24688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:30.696567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:35079 (local address 127.0.0.1:41832)
0806 02:51:30.696735 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:30.696818 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:35079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260806 02:51:30.697193 24685 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.124:35079: Network error: Client connection negotiation failed: client connection to 127.1.210.124:35079: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.124.keytab.
Entry for principal kudu/127.1.210.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.124.keytab.
I20260806 02:51:30.715634 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.124:35079
--webserver_interface=127.1.210.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.124.keytab
--principal=kudu/127.1.210.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:43447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:37905,127.1.210.125:41551,127.1.210.124:35079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:51:30.746577 24685 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.124" port: 35079 } attempt: 1
I20260806 02:51:30.748142 24688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:30.747468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:35079 (local address 127.0.0.1:41842)
0806 02:51:30.747765 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:30.747912 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:35079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260806 02:51:30.749533 24688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:30.748990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:35079 (local address 127.0.0.1:41850)
0806 02:51:30.749216 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:30.749339 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:35079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 02:51:30.749850 24685 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.124:35079: Network error: Client connection negotiation failed: client connection to 127.1.210.124:35079: connect: Connection refused (error 111)
I20260806 02:51:30.784202 24610 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.125" port: 41551 } attempt: 5
Aug 06 02:51:30 dist-test-slave-8tdl krb5kdc[24529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984690, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for kudu/127.1.210.125@KRBTEST.COM
I20260806 02:51:30.800459 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:30.784967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:41551 (local address 127.0.0.1:39576)
0806 02:51:30.785218 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:30.785242 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:30.785378 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:30.785960 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:30.785966 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:30.786480 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:30.786743 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:30.786760 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:30.787889 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:30.787896 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:30.788920 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:30.788940 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:30.790331 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:30.790336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:30.790534 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:30.791305 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:30.791325 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:30.793612 (+ 2287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:30.797864 (+ 4252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:30.797868 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:30.797871 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:30.798308 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:30.798779 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:30.798783 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:30.798785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:30.798926 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:30.799637 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:30.799641 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:30.799793 (+ 152us) client_negotiation.cc:769] Sending connection context
0806 02:51:30.799937 (+ 144us) client_negotiation.cc:240] Negotiation successful
0806 02:51:30.800198 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260806 02:51:30.800698 24690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:30.785078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39576 (local address 127.1.210.125:41551)
0806 02:51:30.785327 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:30.785332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:30.785355 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:51:30.785579 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:30.785587 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:30.785666 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:30.785920 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:30.786924 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:30.787730 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:30.789146 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:30.790135 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:30.790331 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:30.793845 (+ 3514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:30.793853 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:30.793858 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:30.793902 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:30.797690 (+ 3788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:30.798438 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:30.798444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:30.798449 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:30.798631 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:30.799041 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:30.799046 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:30.799050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:30.799454 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:30.799660 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:30.800098 (+ 438us) server_negotiation.cc:299] Negotiation successful
0806 02:51:30.800350 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 02:51:30.801806 24610 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.124" port: 35079 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:51:30.803043 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:30.802551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:35079 (local address 127.0.0.1:41852)
0806 02:51:30.802766 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:30.802862 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:35079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 02:51:30.804080 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:30.803729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:35079 (local address 127.0.0.1:41856)
0806 02:51:30.803886 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:30.803981 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:35079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260806 02:51:30.804301 24610 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.124:35079: Network error: Client connection negotiation failed: client connection to 127.1.210.124:35079: connect: Connection refused (error 111)
I20260806 02:51:30.818097 24685 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.124" port: 35079 } attempt: 2
I20260806 02:51:30.819537 24688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:30.818917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:35079 (local address 127.0.0.1:41866)
0806 02:51:30.819149 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:30.819273 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:35079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260806 02:51:30.820760 24688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:30.820280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:35079 (local address 127.0.0.1:41874)
0806 02:51:30.820452 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:30.820578 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:35079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 02:51:30.821051 24685 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.124:35079: Network error: Client connection negotiation failed: client connection to 127.1.210.124:35079: connect: Connection refused (error 111)
I20260806 02:51:30.855562 24610 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.124" port: 35079 } attempt: 1
I20260806 02:51:30.856889 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:30.856367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:35079 (local address 127.0.0.1:41880)
0806 02:51:30.856616 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:30.856713 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:35079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260806 02:51:30.858045 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:30.857695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:35079 (local address 127.0.0.1:41896)
0806 02:51:30.857843 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:30.857927 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:35079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 02:51:30.858361 24610 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.124:35079: Network error: Client connection negotiation failed: client connection to 127.1.210.124:35079: connect: Connection refused (error 111)
I20260806 02:51:30.912338 24685 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.124" port: 35079 } attempt: 3
I20260806 02:51:30.913887 24688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:30.913187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:35079 (local address 127.0.0.1:41900)
0806 02:51:30.913476 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:30.913634 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:35079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260806 02:51:30.915525 24688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:30.914915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:35079 (local address 127.0.0.1:41916)
0806 02:51:30.915242 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:30.915357 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:35079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213}
W20260806 02:51:30.915789 24685 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.124:35079: Network error: Client connection negotiation failed: client connection to 127.1.210.124:35079: connect: Connection refused (error 111)
I20260806 02:51:30.926658 24610 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.124" port: 35079 } attempt: 2
I20260806 02:51:30.928082 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:30.927525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:35079 (local address 127.0.0.1:41918)
0806 02:51:30.927758 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:30.927855 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:35079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 02:51:30.929373 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:30.928963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:35079 (local address 127.0.0.1:41934)
0806 02:51:30.929169 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:30.929247 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:35079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260806 02:51:30.929615 24610 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.124:35079: Network error: Client connection negotiation failed: client connection to 127.1.210.124:35079: connect: Connection refused (error 111)
I20260806 02:51:31.035845 24610 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.124" port: 35079 } attempt: 3
I20260806 02:51:31.037159 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:31.036537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:35079 (local address 127.0.0.1:41950)
0806 02:51:31.036800 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:31.036924 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:35079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260806 02:51:31.038458 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:31.037976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:35079 (local address 127.0.0.1:41960)
0806 02:51:31.038200 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:31.038294 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:35079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260806 02:51:31.038736 24610 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.124:35079: Network error: Client connection negotiation failed: client connection to 127.1.210.124:35079: connect: Connection refused (error 111)
I20260806 02:51:31.059051 24685 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.124" port: 35079 } attempt: 4
I20260806 02:51:31.060384 24688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:31.059757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:35079 (local address 127.0.0.1:41964)
0806 02:51:31.060045 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:31.060165 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:35079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260806 02:51:31.061637 24688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:31.061213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:35079 (local address 127.0.0.1:41966)
0806 02:51:31.061393 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:31.061488 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:35079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260806 02:51:31.062059 24685 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.124:35079: Network error: Client connection negotiation failed: client connection to 127.1.210.124:35079: connect: Connection refused (error 111)
W20260806 02:51:31.097282 24693 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 02:51:31.097678 24693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:31.097810 24693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:31.097950 24693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:31.105851 24693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:51:31.106000 24693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:31.106137 24693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:31.106227 24693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:51:31.106318 24693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:51:31.116923 24693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:43447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:37905,127.1.210.125:41551,127.1.210.124:35079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.124.keytab
--principal=kudu/127.1.210.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.124:35079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.210.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:51:31.118431 24693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:31.119832 24693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:31.132364 24698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:31.132328 24699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:31.132282 24701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:31.133056 24693 server_base.cc:1034] running on GCE node
I20260806 02:51:31.133930 24693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:31.135540 24693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:31.136831 24693 hybrid_clock.cc:648] HybridClock initialized: now 1785984691136767 us; error 37 us; skew 500 ppm
I20260806 02:51:31.137212 24693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:51:31 dist-test-slave-8tdl krb5kdc[24529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984691, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:31.140813 24693 init.cc:377] Logged in from keytab as kudu/127.1.210.124@KRBTEST.COM (short username kudu)
I20260806 02:51:31.142464 24693 webserver.cc:460] Webserver started at http://127.1.210.124:40079/ using document root <none> and password file <none>
I20260806 02:51:31.143271 24693 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:31.143410 24693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:31.143805 24693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:31.148306 24693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-2/data/instance:
uuid: "0e21397d04f64f9c945d52031d127d17"
format_stamp: "Formatted at 2026-08-06 02:51:31 on dist-test-slave-8tdl"
server_key: "0bda041d18690391b7d5876494ea61fb"
server_key_iv: "6a7f2b491b971ad7a668db2e94d6cf32"
server_key_version: "encryptionkey@0"
I20260806 02:51:31.149149 24693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-2/wal/instance:
uuid: "0e21397d04f64f9c945d52031d127d17"
format_stamp: "Formatted at 2026-08-06 02:51:31 on dist-test-slave-8tdl"
server_key: "0bda041d18690391b7d5876494ea61fb"
server_key_iv: "6a7f2b491b971ad7a668db2e94d6cf32"
server_key_version: "encryptionkey@0"
I20260806 02:51:31.155117 24693 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260806 02:51:31.159021 24708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:31.160512 24693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:51:31.160671 24693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-2/wal
uuid: "0e21397d04f64f9c945d52031d127d17"
format_stamp: "Formatted at 2026-08-06 02:51:31 on dist-test-slave-8tdl"
server_key: "0bda041d18690391b7d5876494ea61fb"
server_key_iv: "6a7f2b491b971ad7a668db2e94d6cf32"
server_key_version: "encryptionkey@0"
I20260806 02:51:31.160873 24693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:31.184844 24693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:31.189757 24693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:31.190085 24693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:31.210317 24693 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.124:35079
I20260806 02:51:31.210340 24759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.124:35079 every 8 connection(s)
I20260806 02:51:31.211845 24693 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-2/data/info.pb
I20260806 02:51:31.215969 24610 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.124" port: 35079 } attempt: 4
I20260806 02:51:31.216475 24760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:31.217757 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 24693
I20260806 02:51:31.217971 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984304136196-1865-0/minicluster-data/master-2/wal/instance
I20260806 02:51:31.218875 1865 external_mini_cluster.cc:1428] Setting key 21f02e37324329bb9dffad4ebec04bd1
I20260806 02:51:31.228209 24760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.126" port: 37905 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:51:31 dist-test-slave-8tdl krb5kdc[24529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984690, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for kudu/127.1.210.124@KRBTEST.COM
Aug 06 02:51:31 dist-test-slave-8tdl krb5kdc[24529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.124@KRBTEST.COM
I20260806 02:51:31.243580 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:31.216733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:35079 (local address 127.0.0.1:41982)
0806 02:51:31.216964 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:31.216993 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:31.217114 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:31.222277 (+ 5163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:31.222283 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:31.222874 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:31.223097 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:31.223110 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.224891 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.224897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:31.226362 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:31.226379 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.227860 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.227866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:31.228088 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:31.228833 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:31.228861 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:31.231506 (+ 2645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:31.238805 (+ 7299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:31.238815 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:31.238822 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:31.239583 (+ 761us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:31.240301 (+ 718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:31.240309 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:31.240315 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:31.240557 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:31.241971 (+ 1414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:31.241979 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:31.242210 (+ 231us) client_negotiation.cc:769] Sending connection context
0806 02:51:31.242389 (+ 179us) client_negotiation.cc:240] Negotiation successful
0806 02:51:31.242966 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260806 02:51:31.247094 24762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:31.217911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41982 (local address 127.1.210.124:35079)
0806 02:51:31.218811 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:31.218823 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:31.218865 (+ 42us) server_negotiation.cc:407] Connection header received
0806 02:51:31.221266 (+ 2401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:31.221295 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:31.221835 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:31.222279 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:31.223276 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.224697 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.226566 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.227667 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.227848 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:31.231864 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:31.231878 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:31.231902 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:31.231968 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:31.238539 (+ 6571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:31.239832 (+ 1293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:31.239845 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:31.239859 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:31.240127 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:31.240769 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:31.240777 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:31.240783 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:31.241748 (+ 965us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:31.244081 (+ 2333us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:31.244483 (+ 402us) server_negotiation.cc:299] Negotiation successful
0806 02:51:31.245139 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":86,"server-negotiator.queue_time_us":651,"spinlock_wait_cycles":35200,"thread_start_us":391,"threads_started":1}
I20260806 02:51:31.247157 24764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:31.220025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41996 (local address 127.1.210.124:35079)
0806 02:51:31.220951 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:31.220957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:31.220981 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:51:31.221266 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:31.221295 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:31.221835 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:31.222280 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:31.225162 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.226497 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.228098 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.228772 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.228910 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:31.234565 (+ 5655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:31.234575 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:31.234582 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:31.234637 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:31.239962 (+ 5325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:31.241267 (+ 1305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:31.241275 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:31.241281 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:31.241502 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:31.242101 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:31.242108 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:31.242114 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:31.242571 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:31.245786 (+ 3215us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:31.245951 (+ 165us) server_negotiation.cc:299] Negotiation successful
0806 02:51:31.246244 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":801,"spinlock_wait_cycles":12672,"thread_start_us":648,"threads_started":1}
I20260806 02:51:31.255455 24610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12a7d16c7b9742728d9faea61a7ad675: Bootstrap starting.
Aug 06 02:51:31 dist-test-slave-8tdl krb5kdc[24529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984691, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.124@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:51:31.259938 24610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12a7d16c7b9742728d9faea61a7ad675: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:31.261186 24610 log.cc:824] T 00000000000000000000000000000000 P 12a7d16c7b9742728d9faea61a7ad675: Log is configured to *not* fsync() on all Append() calls
Aug 06 02:51:31 dist-test-slave-8tdl krb5kdc[24529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:51:31.265239 24610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12a7d16c7b9742728d9faea61a7ad675: No bootstrap required, opened a new log
I20260806 02:51:31.268085 24765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:31.231638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:37905 (local address 127.0.0.1:44784)
0806 02:51:31.247053 (+ 15415us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:31.247114 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:31.249075 (+ 1961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:31.249486 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:31.249493 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:31.250418 (+ 925us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:31.250737 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:31.250757 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.253038 (+ 2281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.253047 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:31.254614 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:31.254633 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.255994 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.256001 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:31.256214 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:31.257036 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:31.257062 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:31.260023 (+ 2961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:31.264825 (+ 4802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:31.264842 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:31.264852 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:31.265650 (+ 798us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:31.266103 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:31.266109 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:31.266113 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:31.266275 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:31.266917 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:31.266925 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:31.267112 (+ 187us) client_negotiation.cc:769] Sending connection context
0806 02:51:31.267293 (+ 181us) client_negotiation.cc:240] Negotiation successful
0806 02:51:31.267644 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15145,"thread_start_us":16410,"threads_started":1}
I20260806 02:51:31.268100 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:31.231690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44784 (local address 127.1.210.126:37905)
0806 02:51:31.248018 (+ 16328us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:31.248024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:31.248043 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:51:31.249243 (+ 1200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:31.249254 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:31.249317 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:31.249434 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:31.251471 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.252782 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.254809 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.255581 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.255723 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:31.260344 (+ 4621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:31.260351 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:31.260356 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:31.260404 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:31.264675 (+ 4271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:31.265788 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:31.265797 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:31.265801 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:31.265935 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:31.266401 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:31.266406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:31.266409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:31.266785 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:31.266945 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:31.267410 (+ 465us) server_negotiation.cc:299] Negotiation successful
0806 02:51:31.267639 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16175,"thread_start_us":16030,"threads_started":1}
I20260806 02:51:31.269794 24760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.125" port: 41551 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:51:31.271977 24767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:31.251360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44792 (local address 127.1.210.126:37905)
0806 02:51:31.253395 (+ 2035us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:31.253402 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:31.253425 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:51:31.253471 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:31.253476 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:31.253543 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:31.253647 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:31.255290 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.256252 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.257698 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.258190 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.258325 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:31.263481 (+ 5156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:31.263486 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:31.263489 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:31.263529 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:31.267299 (+ 3770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:31.268241 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:31.268246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:31.268249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:31.268418 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:31.268898 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:31.268903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:31.268906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:31.269575 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:31.271142 (+ 1567us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:31.271265 (+ 123us) server_negotiation.cc:299] Negotiation successful
0806 02:51:31.271521 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1899,"thread_start_us":1701,"threads_started":1}
I20260806 02:51:31.276710 24610 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12a7d16c7b9742728d9faea61a7ad675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a7d16c7b9742728d9faea61a7ad675" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 37905 } } peers { permanent_uuid: "3261570467c847edb61b03507bf07ed8" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 41551 } } peers { permanent_uuid: "0e21397d04f64f9c945d52031d127d17" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 35079 } }
I20260806 02:51:31.277199 24610 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12a7d16c7b9742728d9faea61a7ad675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12a7d16c7b9742728d9faea61a7ad675, State: Initialized, Role: FOLLOWER
I20260806 02:51:31.277738 24610 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12a7d16c7b9742728d9faea61a7ad675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a7d16c7b9742728d9faea61a7ad675" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 37905 } } peers { permanent_uuid: "3261570467c847edb61b03507bf07ed8" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 41551 } } peers { permanent_uuid: "0e21397d04f64f9c945d52031d127d17" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 35079 } }
Aug 06 02:51:31 dist-test-slave-8tdl krb5kdc[24529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984691, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.124@KRBTEST.COM for kudu/127.1.210.125@KRBTEST.COM
I20260806 02:51:31.278784 24770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12a7d16c7b9742728d9faea61a7ad675 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a7d16c7b9742728d9faea61a7ad675" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 37905 } } peers { permanent_uuid: "3261570467c847edb61b03507bf07ed8" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 41551 } } peers { permanent_uuid: "0e21397d04f64f9c945d52031d127d17" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 35079 } } }
I20260806 02:51:31.279157 24770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12a7d16c7b9742728d9faea61a7ad675 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:51:31.280480 24610 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12a7d16c7b9742728d9faea61a7ad675 [sys.catalog]: configured and running, proceeding with master startup.
Aug 06 02:51:31 dist-test-slave-8tdl krb5kdc[24529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.125@KRBTEST.COM
I20260806 02:51:31.287148 24690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:31.270841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39590 (local address 127.1.210.125:41551)
0806 02:51:31.271055 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:31.271061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:31.271079 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:51:31.271191 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:31.271197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:31.271268 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:31.271372 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:31.272073 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.273088 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.274779 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.275434 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.275584 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:31.279510 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:31.279516 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:31.279520 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:31.279563 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:31.283513 (+ 3950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:31.284497 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:31.284503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:31.284507 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:31.284675 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:31.285198 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:31.285203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:31.285206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:31.285588 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:31.285725 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:31.286357 (+ 632us) server_negotiation.cc:299] Negotiation successful
0806 02:51:31.286627 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 02:51:31.288447 24765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:31.270630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:41551 (local address 127.0.0.1:39590)
0806 02:51:31.270957 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:31.270976 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:31.271083 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:31.271395 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:31.271400 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:31.271785 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:31.271917 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:31.271926 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.273242 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.273248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:31.274589 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:31.274602 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.275827 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.275834 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:31.276051 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:31.276740 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:31.276768 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:31.279260 (+ 2492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:31.283692 (+ 4432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:31.283700 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:31.283708 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:31.284340 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:31.284829 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:31.284833 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:31.284838 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:31.285037 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:31.285824 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:31.285832 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:31.286036 (+ 204us) client_negotiation.cc:769] Sending connection context
0806 02:51:31.287692 (+ 1656us) client_negotiation.cc:240] Negotiation successful
0806 02:51:31.288048 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183}
I20260806 02:51:31.289686 24760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.124" port: 35079 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:51:31.296888 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:31.274093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39598 (local address 127.1.210.125:41551)
0806 02:51:31.274795 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:31.274801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:31.274828 (+ 27us) server_negotiation.cc:407] Connection header received
0806 02:51:31.274913 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:31.274920 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:31.274995 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:31.275104 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:31.276868 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.278003 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.280304 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.281051 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.281189 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:31.287410 (+ 6221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:31.287416 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:31.287420 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:31.287464 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:31.292280 (+ 4816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:31.293490 (+ 1210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:31.293499 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:31.293505 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:31.293695 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:31.294281 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:31.294290 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:31.294295 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:31.294779 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:31.295126 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:31.295893 (+ 767us) server_negotiation.cc:299] Negotiation successful
0806 02:51:31.296243 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":277,"threads_started":1}
Aug 06 02:51:31 dist-test-slave-8tdl krb5kdc[24529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984691, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.124@KRBTEST.COM for kudu/127.1.210.124@KRBTEST.COM
W20260806 02:51:31.303889 24563 tablet.cc:2369] T 00000000000000000000000000000000 P 12a7d16c7b9742728d9faea61a7ad675: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:51:31.304766 1865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:51:31.292569 24542 master_runner.cc:186] Error getting master registration for 127.1.210.125:41551: OK, instance_id { permanent_uuid: "3261570467c847edb61b03507bf07ed8" instance_seqno: 1785984690621373 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 02:51:31.305778 24542 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 02:51:31.306813 24762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:31.290966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42002 (local address 127.1.210.124:35079)
0806 02:51:31.291129 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:31.291134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:31.291147 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:51:31.291181 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:31.291184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:31.291231 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:31.291339 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:31.292206 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.293316 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.295063 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.295719 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.295814 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:31.299930 (+ 4116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:31.299937 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:31.299942 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:31.299980 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:31.303623 (+ 3643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:31.304428 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:31.304433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:31.304436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:31.304608 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:31.305014 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:31.305018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:31.305021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:31.305428 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:31.305584 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:31.306169 (+ 585us) server_negotiation.cc:299] Negotiation successful
0806 02:51:31.306423 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:51:31.307664 24765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:31.290429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:35079 (local address 127.0.0.1:42002)
0806 02:51:31.290818 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:31.290838 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:31.290954 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:31.291354 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:31.291358 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:31.291838 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:31.292053 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:31.292065 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.293475 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.293481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:31.294897 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:31.294915 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.296049 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.296055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:31.296230 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:31.296874 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:31.296897 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:31.299735 (+ 2838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:31.303769 (+ 4034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:31.303774 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:31.303778 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:31.304298 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:31.304741 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:31.304746 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:31.304749 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:31.304894 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:31.305664 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:31.305668 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:31.305836 (+ 168us) client_negotiation.cc:769] Sending connection context
0806 02:51:31.307007 (+ 1171us) client_negotiation.cc:240] Negotiation successful
0806 02:51:31.307315 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256}
I20260806 02:51:31.307852 24585 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "12a7d16c7b9742728d9faea61a7ad675"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44792
I20260806 02:51:31.308449 24585 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12a7d16c7b9742728d9faea61a7ad675 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 02:51:31.308674 24585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12a7d16c7b9742728d9faea61a7ad675 [term 0 FOLLOWER]: Advancing to term 1
W20260806 02:51:31.310469 24781 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 12a7d16c7b9742728d9faea61a7ad675: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:51:31.310714 24781 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:51:31.312085 24585 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12a7d16c7b9742728d9faea61a7ad675 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a7d16c7b9742728d9faea61a7ad675" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 37905 } } peers { permanent_uuid: "3261570467c847edb61b03507bf07ed8" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 41551 } } peers { permanent_uuid: "0e21397d04f64f9c945d52031d127d17" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 35079 } }
I20260806 02:51:31.313603 24760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e21397d04f64f9c945d52031d127d17: Bootstrap starting.
I20260806 02:51:31.314224 24585 leader_election.cc:288] T 00000000000000000000000000000000 P 12a7d16c7b9742728d9faea61a7ad675 [CANDIDATE]: Term 1 election: Requested vote from peers 3261570467c847edb61b03507bf07ed8 (127.1.210.125:41551), 0e21397d04f64f9c945d52031d127d17 (127.1.210.124:35079)
I20260806 02:51:31.318657 24760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e21397d04f64f9c945d52031d127d17: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:31.320134 24760 log.cc:824] T 00000000000000000000000000000000 P 0e21397d04f64f9c945d52031d127d17: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:31.324663 24760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e21397d04f64f9c945d52031d127d17: No bootstrap required, opened a new log
I20260806 02:51:31.337541 24760 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e21397d04f64f9c945d52031d127d17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a7d16c7b9742728d9faea61a7ad675" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 37905 } } peers { permanent_uuid: "3261570467c847edb61b03507bf07ed8" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 41551 } } peers { permanent_uuid: "0e21397d04f64f9c945d52031d127d17" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 35079 } }
I20260806 02:51:31.338097 24760 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e21397d04f64f9c945d52031d127d17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e21397d04f64f9c945d52031d127d17, State: Initialized, Role: FOLLOWER
I20260806 02:51:31.338750 24760 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e21397d04f64f9c945d52031d127d17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a7d16c7b9742728d9faea61a7ad675" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 37905 } } peers { permanent_uuid: "3261570467c847edb61b03507bf07ed8" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 41551 } } peers { permanent_uuid: "0e21397d04f64f9c945d52031d127d17" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 35079 } }
I20260806 02:51:31.341691 24760 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e21397d04f64f9c945d52031d127d17 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:51:31.341742 24735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "12a7d16c7b9742728d9faea61a7ad675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0e21397d04f64f9c945d52031d127d17"
I20260806 02:51:31.342403 24735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e21397d04f64f9c945d52031d127d17 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:31.347913 24735 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0e21397d04f64f9c945d52031d127d17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12a7d16c7b9742728d9faea61a7ad675 in term 1.
I20260806 02:51:31.349344 24560 leader_election.cc:302] T 00000000000000000000000000000000 P 12a7d16c7b9742728d9faea61a7ad675 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e21397d04f64f9c945d52031d127d17, 12a7d16c7b9742728d9faea61a7ad675; no voters:
I20260806 02:51:31.350324 24770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12a7d16c7b9742728d9faea61a7ad675 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:51:31.351426 24770 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12a7d16c7b9742728d9faea61a7ad675 [term 1 LEADER]: Becoming Leader. State: Replica: 12a7d16c7b9742728d9faea61a7ad675, State: Running, Role: LEADER
I20260806 02:51:31.352248 24770 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12a7d16c7b9742728d9faea61a7ad675 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a7d16c7b9742728d9faea61a7ad675" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 37905 } } peers { permanent_uuid: "3261570467c847edb61b03507bf07ed8" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 41551 } } peers { permanent_uuid: "0e21397d04f64f9c945d52031d127d17" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 35079 } }
I20260806 02:51:31.340726 24787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e21397d04f64f9c945d52031d127d17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a7d16c7b9742728d9faea61a7ad675" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 37905 } } peers { permanent_uuid: "3261570467c847edb61b03507bf07ed8" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 41551 } } peers { permanent_uuid: "0e21397d04f64f9c945d52031d127d17" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 35079 } } }
I20260806 02:51:31.357499 24787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e21397d04f64f9c945d52031d127d17 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:51:31.361335 24685 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.124" port: 35079 } attempt: 5
I20260806 02:51:31.363673 24790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12a7d16c7b9742728d9faea61a7ad675 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12a7d16c7b9742728d9faea61a7ad675. Latest consensus state: current_term: 1 leader_uuid: "12a7d16c7b9742728d9faea61a7ad675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a7d16c7b9742728d9faea61a7ad675" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 37905 } } peers { permanent_uuid: "3261570467c847edb61b03507bf07ed8" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 41551 } } peers { permanent_uuid: "0e21397d04f64f9c945d52031d127d17" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 35079 } } }
I20260806 02:51:31.364182 24790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12a7d16c7b9742728d9faea61a7ad675 [sys.catalog]: This master's current role is: LEADER
Aug 06 02:51:31 dist-test-slave-8tdl krb5kdc[24529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984690, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.125@KRBTEST.COM for kudu/127.1.210.124@KRBTEST.COM
I20260806 02:51:31.381640 24764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:31.362498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42016 (local address 127.1.210.124:35079)
0806 02:51:31.362712 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:31.362720 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:31.362741 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:51:31.362805 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:31.362810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:31.362888 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:31.362987 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:31.364190 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.365288 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.367078 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.367742 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.367872 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:31.372410 (+ 4538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:31.372420 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:31.372425 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:31.372490 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:31.376459 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:31.377507 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:31.377515 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:31.377519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:31.377672 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:31.378155 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:31.378163 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:31.378166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:31.378867 (+ 701us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:31.379036 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:31.380345 (+ 1309us) server_negotiation.cc:299] Negotiation successful
0806 02:51:31.380723 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 02:51:31.381865 24688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:31.362108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:35079 (local address 127.0.0.1:42016)
0806 02:51:31.362261 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:31.362281 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:31.362393 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:31.363208 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:31.363212 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:31.363761 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:31.363995 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:31.364010 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.365464 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.365469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:31.366895 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:31.366912 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.368090 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.368096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:31.368339 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:31.369078 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:31.369108 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:31.372154 (+ 3046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:31.376694 (+ 4540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:31.376702 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:31.376705 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:31.377334 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:31.377818 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:31.377825 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:31.377829 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:31.378009 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:31.379105 (+ 1096us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:31.379111 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:31.379293 (+ 182us) client_negotiation.cc:769] Sending connection context
0806 02:51:31.379737 (+ 444us) client_negotiation.cc:240] Negotiation successful
0806 02:51:31.380064 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260806 02:51:31.385978 24767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:31.358510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44794 (local address 127.1.210.126:37905)
0806 02:51:31.358779 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:31.358785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:31.359069 (+ 284us) server_negotiation.cc:407] Connection header received
0806 02:51:31.359555 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:31.359565 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:31.359675 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:31.359770 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:31.365411 (+ 5641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.366578 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.369868 (+ 3290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.370648 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.370796 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:31.377158 (+ 6362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:31.377167 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:31.377172 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:31.377220 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:31.381226 (+ 4006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:31.382741 (+ 1515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:31.382747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:31.382750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:31.382918 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:31.383673 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:31.383678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:31.383682 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:31.384117 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:31.384366 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:31.385347 (+ 981us) server_negotiation.cc:299] Negotiation successful
0806 02:51:31.385593 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 02:51:31.388744 24685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3261570467c847edb61b03507bf07ed8: Bootstrap starting.
I20260806 02:51:31.393272 24685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3261570467c847edb61b03507bf07ed8: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:31.395126 24685 log.cc:824] T 00000000000000000000000000000000 P 3261570467c847edb61b03507bf07ed8: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:31.396338 24796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:51:31.399489 24796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:51:31.412434 24685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3261570467c847edb61b03507bf07ed8: No bootstrap required, opened a new log
I20260806 02:51:31.415925 24762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:31.357841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42006 (local address 127.1.210.124:35079)
0806 02:51:31.358079 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:31.358088 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:31.384979 (+ 26891us) server_negotiation.cc:407] Connection header received
0806 02:51:31.387088 (+ 2109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:31.387095 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:31.387180 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:31.387256 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:31.395930 (+ 8674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.397368 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.400474 (+ 3106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.401420 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.401548 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:31.405343 (+ 3795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:31.405350 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:31.405354 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:31.405402 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:31.409502 (+ 4100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:31.411217 (+ 1715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:31.411249 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:31.411253 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:31.411545 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:31.412342 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:31.412347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:31.412351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:31.413061 (+ 710us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:31.413658 (+ 597us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:31.414733 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0806 02:51:31.415247 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 02:51:31.421731 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:31.372353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39612 (local address 127.1.210.125:41551)
0806 02:51:31.372559 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:31.372566 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:31.387409 (+ 14843us) server_negotiation.cc:407] Connection header received
0806 02:51:31.387726 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:31.387731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:31.387822 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:31.387892 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:31.390905 (+ 3013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.392041 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.395024 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.395703 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.395828 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:31.402100 (+ 6272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:31.402107 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:31.402112 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:31.402159 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:31.406040 (+ 3881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:31.418456 (+ 12416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:31.418463 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:31.418467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:31.418639 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:31.419441 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:31.419446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:31.419450 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:31.419861 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:31.420133 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:31.421070 (+ 937us) server_negotiation.cc:299] Negotiation successful
0806 02:51:31.421353 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 02:51:31.430944 24685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3261570467c847edb61b03507bf07ed8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a7d16c7b9742728d9faea61a7ad675" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 37905 } } peers { permanent_uuid: "3261570467c847edb61b03507bf07ed8" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 41551 } } peers { permanent_uuid: "0e21397d04f64f9c945d52031d127d17" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 35079 } }
I20260806 02:51:31.432683 24735 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0e21397d04f64f9c945d52031d127d17 [term 1 FOLLOWER]: Refusing update from remote peer 12a7d16c7b9742728d9faea61a7ad675: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:51:31.433815 24790 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 12a7d16c7b9742728d9faea61a7ad675 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e21397d04f64f9c945d52031d127d17" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 35079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:51:31.434772 24685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3261570467c847edb61b03507bf07ed8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3261570467c847edb61b03507bf07ed8, State: Initialized, Role: FOLLOWER
I20260806 02:51:31.441080 24685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3261570467c847edb61b03507bf07ed8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a7d16c7b9742728d9faea61a7ad675" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 37905 } } peers { permanent_uuid: "3261570467c847edb61b03507bf07ed8" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 41551 } } peers { permanent_uuid: "0e21397d04f64f9c945d52031d127d17" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 35079 } }
W20260806 02:51:31.442581 24714 tablet.cc:2369] T 00000000000000000000000000000000 P 0e21397d04f64f9c945d52031d127d17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:51:31.444631 24685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3261570467c847edb61b03507bf07ed8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:51:31.445089 24659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3261570467c847edb61b03507bf07ed8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:31.451628 24659 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3261570467c847edb61b03507bf07ed8 [term 1 FOLLOWER]: Refusing update from remote peer 12a7d16c7b9742728d9faea61a7ad675: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:51:31.452929 24790 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 12a7d16c7b9742728d9faea61a7ad675 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3261570467c847edb61b03507bf07ed8" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 41551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:51:31.445281 24660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "12a7d16c7b9742728d9faea61a7ad675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3261570467c847edb61b03507bf07ed8"
I20260806 02:51:31.457784 24660 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3261570467c847edb61b03507bf07ed8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12a7d16c7b9742728d9faea61a7ad675 in term 1.
I20260806 02:51:31.444149 24805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3261570467c847edb61b03507bf07ed8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a7d16c7b9742728d9faea61a7ad675" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 37905 } } peers { permanent_uuid: "3261570467c847edb61b03507bf07ed8" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 41551 } } peers { permanent_uuid: "0e21397d04f64f9c945d52031d127d17" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 35079 } } }
I20260806 02:51:31.461237 24805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3261570467c847edb61b03507bf07ed8 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:51:31.463742 24787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e21397d04f64f9c945d52031d127d17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12a7d16c7b9742728d9faea61a7ad675. Latest consensus state: current_term: 1 leader_uuid: "12a7d16c7b9742728d9faea61a7ad675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a7d16c7b9742728d9faea61a7ad675" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 37905 } } peers { permanent_uuid: "3261570467c847edb61b03507bf07ed8" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 41551 } } peers { permanent_uuid: "0e21397d04f64f9c945d52031d127d17" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 35079 } } }
I20260806 02:51:31.464154 24787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e21397d04f64f9c945d52031d127d17 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:51:31.466547 24801 mvcc.cc:205] Tried to move back new op lower bound from 7315393296080760832 to 7315393295787540480. Current Snapshot: MvccSnapshot[applied={T|T < 7315393296080760832}]
I20260806 02:51:31.471994 24787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e21397d04f64f9c945d52031d127d17 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "12a7d16c7b9742728d9faea61a7ad675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a7d16c7b9742728d9faea61a7ad675" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 37905 } } peers { permanent_uuid: "3261570467c847edb61b03507bf07ed8" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 41551 } } peers { permanent_uuid: "0e21397d04f64f9c945d52031d127d17" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 35079 } } }
I20260806 02:51:31.472288 24787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e21397d04f64f9c945d52031d127d17 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:51:31.477567 24770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12a7d16c7b9742728d9faea61a7ad675 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "12a7d16c7b9742728d9faea61a7ad675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a7d16c7b9742728d9faea61a7ad675" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 37905 } } peers { permanent_uuid: "3261570467c847edb61b03507bf07ed8" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 41551 } } peers { permanent_uuid: "0e21397d04f64f9c945d52031d127d17" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 35079 } } }
I20260806 02:51:31.477921 24770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12a7d16c7b9742728d9faea61a7ad675 [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:31.479481 24796 catalog_manager.cc:1347] Generated new cluster ID: 5197831afef54e3bad2dc1c3015cf46e
I20260806 02:51:31.479602 24796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:51:31.481802 24815 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 02:51:31.494930 24796 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:51:31.496723 24796 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:51:31.487860 24693 master_runner.cc:186] Error getting master registration for 127.1.210.125:41551: OK, instance_id { permanent_uuid: "3261570467c847edb61b03507bf07ed8" instance_seqno: 1785984690621373 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 02:51:31.498411 24693 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 02:51:31.502298 24805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3261570467c847edb61b03507bf07ed8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12a7d16c7b9742728d9faea61a7ad675. Latest consensus state: current_term: 1 leader_uuid: "12a7d16c7b9742728d9faea61a7ad675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a7d16c7b9742728d9faea61a7ad675" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 37905 } } peers { permanent_uuid: "3261570467c847edb61b03507bf07ed8" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 41551 } } peers { permanent_uuid: "0e21397d04f64f9c945d52031d127d17" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 35079 } } }
I20260806 02:51:31.502782 24805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3261570467c847edb61b03507bf07ed8 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:51:31.507022 24770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12a7d16c7b9742728d9faea61a7ad675 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "12a7d16c7b9742728d9faea61a7ad675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a7d16c7b9742728d9faea61a7ad675" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 37905 } } peers { permanent_uuid: "3261570467c847edb61b03507bf07ed8" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 41551 } } peers { permanent_uuid: "0e21397d04f64f9c945d52031d127d17" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 35079 } } }
I20260806 02:51:31.507439 24770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12a7d16c7b9742728d9faea61a7ad675 [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:31.523314 24805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3261570467c847edb61b03507bf07ed8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "12a7d16c7b9742728d9faea61a7ad675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a7d16c7b9742728d9faea61a7ad675" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 37905 } } peers { permanent_uuid: "3261570467c847edb61b03507bf07ed8" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 41551 } } peers { permanent_uuid: "0e21397d04f64f9c945d52031d127d17" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 35079 } } }
I20260806 02:51:31.523599 24805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3261570467c847edb61b03507bf07ed8 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:51:31.526736 24796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12a7d16c7b9742728d9faea61a7ad675: Generated new TSK 0
I20260806 02:51:31.527863 24796 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:51:31.535791 24831 catalog_manager.cc:1259] Loaded cluster ID: 5197831afef54e3bad2dc1c3015cf46e
I20260806 02:51:31.535980 24831 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3261570467c847edb61b03507bf07ed8: loading cluster ID for follower catalog manager: success
I20260806 02:51:31.539841 24831 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3261570467c847edb61b03507bf07ed8: acquiring CA information for follower catalog manager: success
I20260806 02:51:31.542480 24831 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3261570467c847edb61b03507bf07ed8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 02:51:31.652180 24767 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:31.625839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44808 (local address 127.1.210.126:37905)
0806 02:51:31.626081 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:31.626090 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:31.649776 (+ 23686us) server_negotiation.cc:407] Connection header received
0806 02:51:31.650357 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:31.650365 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:31.650462 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:31.650552 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:31.651870 (+ 1318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44808: BlockingRecv error: recv got EOF from 127.0.0.1:44808 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260806 02:51:31.655340 24762 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:31.626039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42022 (local address 127.1.210.124:35079)
0806 02:51:31.626302 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:31.626310 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:31.653635 (+ 27325us) server_negotiation.cc:407] Connection header received
0806 02:51:31.653863 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:31.653870 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:31.653968 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:31.654055 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:31.655095 (+ 1040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42022: BlockingRecv error: recv got EOF from 127.0.0.1:42022 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260806 02:51:31.657514 24769 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:31.624914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39626 (local address 127.1.210.125:41551)
0806 02:51:31.625097 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:31.625105 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:31.653157 (+ 28052us) server_negotiation.cc:407] Connection header received
0806 02:51:31.655856 (+ 2699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:31.655862 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:31.655976 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:31.656064 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:31.657242 (+ 1178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39626: BlockingRecv error: recv got EOF from 127.0.0.1:39626 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 02:51:31.676615 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:31.667812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39640 (local address 127.1.210.125:41551)
0806 02:51:31.668004 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:31.668010 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:31.668511 (+ 501us) server_negotiation.cc:407] Connection header received
0806 02:51:31.668763 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:31.668769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:31.669020 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:31.669209 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:51:31.669916 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.671150 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.673454 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.674190 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.674331 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:31.675040 (+ 709us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:51:31.675816 (+ 776us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:51:31.675987 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:31.676141 (+ 154us) server_negotiation.cc:299] Negotiation successful
0806 02:51:31.676244 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260806 02:51:31.689442 24762 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:31.686784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42026 (local address 127.1.210.124:35079)
0806 02:51:31.687015 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:31.687022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:31.687651 (+ 629us) server_negotiation.cc:407] Connection header received
0806 02:51:31.687908 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:31.687915 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:31.688049 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:31.688157 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:31.689197 (+ 1040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42026: BlockingRecv error: recv got EOF from 127.0.0.1:42026 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 02:51:31.702095 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:31.694403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39646 (local address 127.1.210.125:41551)
0806 02:51:31.694595 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:31.694606 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:31.695191 (+ 585us) server_negotiation.cc:407] Connection header received
0806 02:51:31.695389 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:31.695396 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:31.695504 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:31.695615 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:51:31.696090 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.697396 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.699474 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.700334 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.700507 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:31.701067 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:51:31.701249 (+ 182us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:51:31.701397 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:31.701601 (+ 204us) server_negotiation.cc:299] Negotiation successful
0806 02:51:31.701702 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260806 02:51:31.713076 24762 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:31.711049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42034 (local address 127.1.210.124:35079)
0806 02:51:31.711232 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:31.711238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:31.711413 (+ 175us) server_negotiation.cc:407] Connection header received
0806 02:51:31.711721 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:31.711727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:31.711817 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:31.711919 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:31.712824 (+ 905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42034: BlockingRecv error: recv got EOF from 127.0.0.1:42034 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 02:51:31.733765 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:31.718656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39650 (local address 127.1.210.125:41551)
0806 02:51:31.718908 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:31.718915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:31.719616 (+ 701us) server_negotiation.cc:407] Connection header received
0806 02:51:31.719933 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:31.719953 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:31.720052 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:31.720120 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:51:31.726459 (+ 6339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.727834 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.731076 (+ 3242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.731831 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.731958 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:31.732765 (+ 807us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:51:31.732959 (+ 194us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:51:31.733043 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:31.733282 (+ 239us) server_negotiation.cc:299] Negotiation successful
0806 02:51:31.733395 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
W20260806 02:51:31.750887 24762 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:31.748401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42040 (local address 127.1.210.124:35079)
0806 02:51:31.748596 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:31.748604 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:31.749187 (+ 583us) server_negotiation.cc:407] Connection header received
0806 02:51:31.749384 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:31.749391 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:31.749498 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:31.749619 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:31.750549 (+ 930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42040: BlockingRecv error: recv got EOF from 127.0.0.1:42040 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 02:51:31.766265 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:31.758466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39662 (local address 127.1.210.125:41551)
0806 02:51:31.758742 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:31.758749 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:31.759312 (+ 563us) server_negotiation.cc:407] Connection header received
0806 02:51:31.759516 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:31.759523 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:31.759626 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:31.759738 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:51:31.760232 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.761576 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.763605 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.764391 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.764565 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:31.765150 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:51:31.765357 (+ 207us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:51:31.765461 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:31.765647 (+ 186us) server_negotiation.cc:299] Negotiation successful
0806 02:51:31.765744 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
W20260806 02:51:31.779449 24762 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:31.776969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42054 (local address 127.1.210.124:35079)
0806 02:51:31.777261 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:31.777268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:31.777865 (+ 597us) server_negotiation.cc:407] Connection header received
0806 02:51:31.778033 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:31.778040 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:31.778135 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:31.778246 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:31.779231 (+ 985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42054: BlockingRecv error: recv got EOF from 127.0.0.1:42054 (error 108)
Metrics: {"server-negotiator.queue_time_us":166}
I20260806 02:51:31.799075 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:31.791289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39670 (local address 127.1.210.125:41551)
0806 02:51:31.791526 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:31.791537 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:31.792090 (+ 553us) server_negotiation.cc:407] Connection header received
0806 02:51:31.792363 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:31.792370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:31.792495 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:31.792565 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:51:31.793247 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.794470 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.796599 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.797424 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.797590 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:31.798100 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:51:31.798254 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:51:31.798355 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:31.798606 (+ 251us) server_negotiation.cc:299] Negotiation successful
0806 02:51:31.798742 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260806 02:51:31.808594 24762 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:31.806159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42060 (local address 127.1.210.124:35079)
0806 02:51:31.806356 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:31.806362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:31.806941 (+ 579us) server_negotiation.cc:407] Connection header received
0806 02:51:31.807168 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:31.807174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:31.807277 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:31.807385 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:31.808312 (+ 927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42060: BlockingRecv error: recv got EOF from 127.0.0.1:42060 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 02:51:31.835928 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:31.828575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39680 (local address 127.1.210.125:41551)
0806 02:51:31.828799 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:31.828809 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:31.829403 (+ 594us) server_negotiation.cc:407] Connection header received
0806 02:51:31.829795 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:31.829802 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:31.829894 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:31.829991 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:51:31.830568 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.831799 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.833797 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.834427 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.834551 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:31.835141 (+ 590us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:51:31.835273 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:51:31.835380 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:31.835561 (+ 181us) server_negotiation.cc:299] Negotiation successful
0806 02:51:31.835642 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260806 02:51:31.851500 24762 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:31.849045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42072 (local address 127.1.210.124:35079)
0806 02:51:31.849265 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:31.849272 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:31.849834 (+ 562us) server_negotiation.cc:407] Connection header received
0806 02:51:31.850059 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:31.850066 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:31.850165 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:31.850278 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:31.851227 (+ 949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42072: BlockingRecv error: recv got EOF from 127.0.0.1:42072 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 02:51:31.894815 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:31.887327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39682 (local address 127.1.210.125:41551)
0806 02:51:31.887556 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:31.887563 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:31.888144 (+ 581us) server_negotiation.cc:407] Connection header received
0806 02:51:31.888363 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:31.888369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:31.888465 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:31.888577 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:51:31.889117 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.890339 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.892393 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.893149 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.893288 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:31.893863 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:51:31.894038 (+ 175us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:51:31.894129 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:31.894346 (+ 217us) server_negotiation.cc:299] Negotiation successful
0806 02:51:31.894457 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260806 02:51:31.904908 24762 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:31.902546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42082 (local address 127.1.210.124:35079)
0806 02:51:31.902756 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:31.902762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:31.903323 (+ 561us) server_negotiation.cc:407] Connection header received
0806 02:51:31.903542 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:31.903548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:31.903648 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:31.903759 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:31.904661 (+ 902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42082: BlockingRecv error: recv got EOF from 127.0.0.1:42082 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 02:51:31.983911 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:31.976802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39692 (local address 127.1.210.125:41551)
0806 02:51:31.977021 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:31.977028 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:31.977590 (+ 562us) server_negotiation.cc:407] Connection header received
0806 02:51:31.977993 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:31.978007 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:31.978106 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:31.978209 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:51:31.978802 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.979778 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.981793 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:31.982399 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:31.982526 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:31.983151 (+ 625us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:51:31.983278 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:51:31.983349 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:31.983497 (+ 148us) server_negotiation.cc:299] Negotiation successful
0806 02:51:31.983566 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260806 02:51:31.993238 24762 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:31.990979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42086 (local address 127.1.210.124:35079)
0806 02:51:31.991191 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:31.991199 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:31.991779 (+ 580us) server_negotiation.cc:407] Connection header received
0806 02:51:31.991974 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:31.991978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:31.992089 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:31.992177 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:31.993067 (+ 890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42086: BlockingRecv error: recv got EOF from 127.0.0.1:42086 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 02:51:32.130801 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:32.124980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39698 (local address 127.1.210.125:41551)
0806 02:51:32.125263 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:32.125269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:32.125559 (+ 290us) server_negotiation.cc:407] Connection header received
0806 02:51:32.125807 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:32.125812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:32.125883 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:32.126030 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:51:32.126500 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.127283 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.128737 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.129324 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.129465 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:32.129921 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:51:32.130056 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:51:32.130131 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:32.130424 (+ 293us) server_negotiation.cc:299] Negotiation successful
0806 02:51:32.130511 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
W20260806 02:51:32.139357 24762 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:32.137226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42090 (local address 127.1.210.124:35079)
0806 02:51:32.137475 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:32.137482 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:32.137732 (+ 250us) server_negotiation.cc:407] Connection header received
0806 02:51:32.137988 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:32.137995 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:32.138089 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:32.138213 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:32.139122 (+ 909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42090: BlockingRecv error: recv got EOF from 127.0.0.1:42090 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 02:51:32.405143 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:32.399801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39700 (local address 127.1.210.125:41551)
0806 02:51:32.400054 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:32.400063 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:32.400321 (+ 258us) server_negotiation.cc:407] Connection header received
0806 02:51:32.400567 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:32.400571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:32.400641 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:32.400755 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:51:32.401248 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.401960 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.403414 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.403826 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.403977 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:32.404387 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:51:32.404499 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:51:32.404598 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:32.404755 (+ 157us) server_negotiation.cc:299] Negotiation successful
0806 02:51:32.404838 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260806 02:51:32.413189 24762 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:32.411622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42094 (local address 127.1.210.124:35079)
0806 02:51:32.411797 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:32.411801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:32.412007 (+ 206us) server_negotiation.cc:407] Connection header received
0806 02:51:32.412116 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:32.412120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:32.412200 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:32.412283 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:32.413014 (+ 731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42094: BlockingRecv error: recv got EOF from 127.0.0.1:42094 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 02:51:32.486637 24815 catalog_manager.cc:1259] Loaded cluster ID: 5197831afef54e3bad2dc1c3015cf46e
I20260806 02:51:32.486799 24815 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0e21397d04f64f9c945d52031d127d17: loading cluster ID for follower catalog manager: success
I20260806 02:51:32.489710 24815 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0e21397d04f64f9c945d52031d127d17: acquiring CA information for follower catalog manager: success
I20260806 02:51:32.491729 24815 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0e21397d04f64f9c945d52031d127d17: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:51:32.934479 24949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:32.928807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39704 (local address 127.1.210.125:41551)
0806 02:51:32.929342 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:32.929346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:32.929473 (+ 127us) server_negotiation.cc:407] Connection header received
0806 02:51:32.929650 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:32.929655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:32.929724 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:32.929814 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:51:32.930281 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.931223 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.932622 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.933096 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.933265 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:32.933700 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:51:32.933821 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:51:32.933923 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:32.934088 (+ 165us) server_negotiation.cc:299] Negotiation successful
0806 02:51:32.934173 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":274,"threads_started":1}
I20260806 02:51:32.947731 24955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:32.941247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42106 (local address 127.1.210.124:35079)
0806 02:51:32.941756 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:32.941761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:32.941849 (+ 88us) server_negotiation.cc:407] Connection header received
0806 02:51:32.942088 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:32.942093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:32.942251 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:32.942366 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:51:32.942982 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.943803 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.945213 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.945657 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.945769 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:32.946432 (+ 663us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:51:32.947177 (+ 745us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:51:32.947299 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:32.947425 (+ 126us) server_negotiation.cc:299] Negotiation successful
0806 02:51:32.947498 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":296,"threads_started":1}
I20260806 02:51:32.968688 24949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:32.955100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39714 (local address 127.1.210.125:41551)
0806 02:51:32.955372 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:32.955381 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:32.961790 (+ 6409us) server_negotiation.cc:407] Connection header received
0806 02:51:32.962004 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:32.962010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:32.962144 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:32.962260 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:51:32.963111 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.963859 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.966803 (+ 2944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.967252 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.967365 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:32.967863 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:51:32.967971 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:51:32.968047 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:32.968253 (+ 206us) server_negotiation.cc:299] Negotiation successful
0806 02:51:32.968343 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:51:32.971386 24962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:32.954635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44822 (local address 127.1.210.126:37905)
0806 02:51:32.962467 (+ 7832us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:32.962475 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:32.962492 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:51:32.962613 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:32.962616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:32.962852 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:32.963047 (+ 195us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:51:32.965097 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.966354 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.968304 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.969163 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.969316 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:32.970110 (+ 794us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:51:32.970660 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:51:32.970820 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:32.970962 (+ 142us) server_negotiation.cc:299] Negotiation successful
0806 02:51:32.971039 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7693,"thread_start_us":7701,"threads_started":1}
I20260806 02:51:32.972218 24955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:32.955362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42116 (local address 127.1.210.124:35079)
0806 02:51:32.955580 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:32.955589 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:32.964969 (+ 9380us) server_negotiation.cc:407] Connection header received
0806 02:51:32.965255 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:32.965261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:32.965376 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:32.965472 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:51:32.966050 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.967035 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.969939 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:32.970520 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:32.970650 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:32.971426 (+ 776us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:51:32.971552 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:51:32.971632 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:32.971876 (+ 244us) server_negotiation.cc:299] Negotiation successful
0806 02:51:32.971954 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 02:51:32.977387 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 24542
I20260806 02:51:32.979179 24605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:51:33.094715 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 24542
I20260806 02:51:33.122160 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 24617
I20260806 02:51:33.123629 24680 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:51:33.238034 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 24617
I20260806 02:51:33.262041 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 24693
I20260806 02:51:33.263342 24755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:51:33.372952 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 24693
2026-08-06T02:51:33Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5293 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:51:33 dist-test-slave-8tdl krb5kdc[24969](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:51:33 dist-test-slave-8tdl krb5kdc[24969](info): set up 2 sockets
Aug 06 02:51:33 dist-test-slave-8tdl krb5kdc[24969](info): commencing operation
krb5kdc: starting...
W20260806 02:51:34.522467 1865 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.001s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/krb5kdc/test-user.keytab.
Aug 06 02:51:34 dist-test-slave-8tdl krb5kdc[24969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:51:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:51:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.210.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
Entry for principal kudu/127.1.210.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab.
I20260806 02:51:34.871814 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:41437
--webserver_interface=127.1.210.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:41437,127.1.210.125:45019,127.1.210.124:41239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:35.234452 24983 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 02:51:35.234892 24983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:35.234968 24983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:35.235054 24983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:35.242564 24983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:51:35.242753 24983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:35.242837 24983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:35.242892 24983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:51:35.242951 24983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:51:35.253902 24983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:41437,127.1.210.125:45019,127.1.210.124:41239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.126.keytab
--principal=kudu/127.1.210.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.126:41437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.210.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:51:35.255357 24983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:35.256731 24983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:35.269261 24988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:35.269296 24991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:35.269198 24989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:35.270617 24983 server_base.cc:1034] running on GCE node
I20260806 02:51:35.271507 24983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:35.273128 24983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:35.274381 24983 hybrid_clock.cc:648] HybridClock initialized: now 1785984695274320 us; error 35 us; skew 500 ppm
I20260806 02:51:35.274823 24983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:51:35 dist-test-slave-8tdl krb5kdc[24969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984695, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:35.284720 24983 init.cc:377] Logged in from keytab as kudu/127.1.210.126@KRBTEST.COM (short username kudu)
I20260806 02:51:35.286362 24983 webserver.cc:460] Webserver started at http://127.1.210.126:39487/ using document root <none> and password file <none>
I20260806 02:51:35.287174 24983 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:35.287324 24983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:35.287708 24983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:35.290979 24983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-0/data/instance:
uuid: "55c0f81eb98548ff80d7374a3843aa91"
format_stamp: "Formatted at 2026-08-06 02:51:35 on dist-test-slave-8tdl"
server_key: "b151aee3fd89f6d3e2f9f5ebf4771730"
server_key_iv: "f6b1107a838d956268f4b98859e56a07"
server_key_version: "encryptionkey@0"
I20260806 02:51:35.291823 24983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-0/wal/instance:
uuid: "55c0f81eb98548ff80d7374a3843aa91"
format_stamp: "Formatted at 2026-08-06 02:51:35 on dist-test-slave-8tdl"
server_key: "b151aee3fd89f6d3e2f9f5ebf4771730"
server_key_iv: "f6b1107a838d956268f4b98859e56a07"
server_key_version: "encryptionkey@0"
I20260806 02:51:35.297324 24983 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 02:51:35.301133 24998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:35.302575 24983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 02:51:35.302791 24983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-0/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-0/wal
uuid: "55c0f81eb98548ff80d7374a3843aa91"
format_stamp: "Formatted at 2026-08-06 02:51:35 on dist-test-slave-8tdl"
server_key: "b151aee3fd89f6d3e2f9f5ebf4771730"
server_key_iv: "f6b1107a838d956268f4b98859e56a07"
server_key_version: "encryptionkey@0"
I20260806 02:51:35.303017 24983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:35.335641 24983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:35.340611 24983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:35.340937 24983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:35.361940 24983 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.126:41437
I20260806 02:51:35.362035 25050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.126:41437 every 8 connection(s)
I20260806 02:51:35.363390 24983 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-0/data/info.pb
I20260806 02:51:35.366431 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 24983
I20260806 02:51:35.366647 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-0/wal/instance
I20260806 02:51:35.367228 1865 external_mini_cluster.cc:1428] Setting key 9b7b84c9d7a3dcf9c8d3dfc1de5d3d1a
I20260806 02:51:35.368192 25051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:35.378744 25051 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41437 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:51:35 dist-test-slave-8tdl krb5kdc[24969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984695, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.210.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.125@KRBTEST.COM" created.
I20260806 02:51:35.404747 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:35.383199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41437 (local address 127.0.0.1:47796)
0806 02:51:35.384103 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:35.384150 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:35.386409 (+ 2259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:35.387328 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:35.387334 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:35.388100 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:35.388478 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:35.388497 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:35.390117 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:35.390122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:35.391391 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:35.391404 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:35.392402 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:35.392409 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:35.392702 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:35.393380 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:35.393408 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:35.396179 (+ 2771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:35.400013 (+ 3834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:35.400027 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:35.400041 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:35.400650 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:35.401212 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:35.401216 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:35.401218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:35.401372 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:35.402332 (+ 960us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:35.402338 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:35.402560 (+ 222us) client_negotiation.cc:769] Sending connection context
0806 02:51:35.402908 (+ 348us) client_negotiation.cc:240] Negotiation successful
0806 02:51:35.403420 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":13,"thread_start_us":403,"threads_started":1}
I20260806 02:51:35.404843 25055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:35.384654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47796 (local address 127.1.210.126:41437)
0806 02:51:35.385457 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:35.385470 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:35.385531 (+ 61us) server_negotiation.cc:407] Connection header received
0806 02:51:35.386835 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:35.386841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:35.387094 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:35.387314 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:35.388704 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:35.389947 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:35.391545 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:35.392159 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:35.392370 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:35.396666 (+ 4296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:35.396687 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:35.396708 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:35.396754 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:35.399853 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:35.400869 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:35.400876 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:35.400921 (+ 45us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:35.401092 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:35.401502 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:35.401506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:35.401508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:35.402178 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:35.402347 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:35.403141 (+ 794us) server_negotiation.cc:299] Negotiation successful
0806 02:51:35.403612 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":233,"server-negotiator.queue_time_us":580,"thread_start_us":357,"threads_started":1}
I20260806 02:51:35.408727 25051 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.125" port: 45019 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:51:35.410090 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:35.409516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:45019 (local address 127.0.0.1:36558)
0806 02:51:35.409745 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:35.409910 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.125:45019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260806 02:51:35.410425 25002 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.210.125:45019: connect: Connection refused (error 111)
I20260806 02:51:35.412351 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:35.411798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:45019 (local address 127.0.0.1:36568)
0806 02:51:35.412050 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:35.412158 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.125:45019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260806 02:51:35.412663 25051 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.125:45019: Network error: Client connection negotiation failed: client connection to 127.1.210.125:45019: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.125.keytab.
Entry for principal kudu/127.1.210.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.125.keytab.
I20260806 02:51:35.445961 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.125:45019
--webserver_interface=127.1.210.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.125.keytab
--principal=kudu/127.1.210.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:41437,127.1.210.125:45019,127.1.210.124:41239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:51:35.462064 25051 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.125" port: 45019 } attempt: 1
I20260806 02:51:35.463665 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:35.462957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:45019 (local address 127.0.0.1:36576)
0806 02:51:35.463261 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:35.463470 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.125:45019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260806 02:51:35.464818 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:35.464370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:45019 (local address 127.0.0.1:36584)
0806 02:51:35.464545 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:35.464641 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.125:45019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 02:51:35.465050 25051 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.125:45019: Network error: Client connection negotiation failed: client connection to 127.1.210.125:45019: connect: Connection refused (error 111)
I20260806 02:51:35.533380 25051 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.125" port: 45019 } attempt: 2
I20260806 02:51:35.535035 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:35.534267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:45019 (local address 127.0.0.1:36586)
0806 02:51:35.534598 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:35.534803 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.125:45019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260806 02:51:35.536288 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:35.535837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:45019 (local address 127.0.0.1:36594)
0806 02:51:35.536014 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:35.536127 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.125:45019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260806 02:51:35.536548 25051 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.125:45019: Network error: Client connection negotiation failed: client connection to 127.1.210.125:45019: connect: Connection refused (error 111)
I20260806 02:51:35.627840 25051 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.125" port: 45019 } attempt: 3
I20260806 02:51:35.629426 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:35.628863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:45019 (local address 127.0.0.1:36606)
0806 02:51:35.629097 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:35.629247 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.125:45019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260806 02:51:35.630626 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:35.630197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:45019 (local address 127.0.0.1:36618)
0806 02:51:35.630372 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:35.630480 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.125:45019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260806 02:51:35.630885 25051 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.125:45019: Network error: Client connection negotiation failed: client connection to 127.1.210.125:45019: connect: Connection refused (error 111)
I20260806 02:51:35.774139 25051 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.125" port: 45019 } attempt: 4
I20260806 02:51:35.775563 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:35.774965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:45019 (local address 127.0.0.1:36632)
0806 02:51:35.775305 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:35.775397 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.125:45019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
I20260806 02:51:35.776707 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:35.776299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:45019 (local address 127.0.0.1:36648)
0806 02:51:35.776492 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:35.776574 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.125:45019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260806 02:51:35.776901 25051 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.125:45019: Network error: Client connection negotiation failed: client connection to 127.1.210.125:45019: connect: Connection refused (error 111)
W20260806 02:51:35.812263 25058 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 02:51:35.812666 25058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:35.812793 25058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:35.812934 25058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:35.820513 25058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:51:35.820650 25058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:35.820786 25058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:35.820873 25058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:51:35.820992 25058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:51:35.831234 25058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:41437,127.1.210.125:45019,127.1.210.124:41239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.125.keytab
--principal=kudu/127.1.210.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.125:45019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.210.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:51:35.832680 25058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:35.834061 25058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:35.846133 25063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:35.846246 25066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:35.846452 25064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:35.847417 25058 server_base.cc:1034] running on GCE node
I20260806 02:51:35.848238 25058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:35.849659 25058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:35.850903 25058 hybrid_clock.cc:648] HybridClock initialized: now 1785984695850781 us; error 93 us; skew 500 ppm
I20260806 02:51:35.851228 25058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:51:35 dist-test-slave-8tdl krb5kdc[24969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984695, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:35.854619 25058 init.cc:377] Logged in from keytab as kudu/127.1.210.125@KRBTEST.COM (short username kudu)
I20260806 02:51:35.856217 25058 webserver.cc:460] Webserver started at http://127.1.210.125:33333/ using document root <none> and password file <none>
I20260806 02:51:35.856949 25058 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:35.857086 25058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:35.857434 25058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:35.860599 25058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-1/data/instance:
uuid: "c44a3437a4fe44c5beb39ad2d0ccdd1f"
format_stamp: "Formatted at 2026-08-06 02:51:35 on dist-test-slave-8tdl"
server_key: "111b120730bffd60abce84f2691040e6"
server_key_iv: "895f3979f81868ddfbc0e036269771b8"
server_key_version: "encryptionkey@0"
I20260806 02:51:35.861434 25058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-1/wal/instance:
uuid: "c44a3437a4fe44c5beb39ad2d0ccdd1f"
format_stamp: "Formatted at 2026-08-06 02:51:35 on dist-test-slave-8tdl"
server_key: "111b120730bffd60abce84f2691040e6"
server_key_iv: "895f3979f81868ddfbc0e036269771b8"
server_key_version: "encryptionkey@0"
I20260806 02:51:35.867177 25058 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260806 02:51:35.870855 25073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:35.872310 25058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:51:35.872519 25058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-1/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-1/wal
uuid: "c44a3437a4fe44c5beb39ad2d0ccdd1f"
format_stamp: "Formatted at 2026-08-06 02:51:35 on dist-test-slave-8tdl"
server_key: "111b120730bffd60abce84f2691040e6"
server_key_iv: "895f3979f81868ddfbc0e036269771b8"
server_key_version: "encryptionkey@0"
I20260806 02:51:35.872748 25058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:35.895939 25058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:35.900336 25058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:35.900629 25058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:35.920462 25058 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.125:45019
I20260806 02:51:35.920500 25126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.125:45019 every 8 connection(s)
I20260806 02:51:35.921875 25058 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-1/data/info.pb
I20260806 02:51:35.926131 25127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:35.931972 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 25058
I20260806 02:51:35.932207 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-1/wal/instance
I20260806 02:51:35.932705 1865 external_mini_cluster.cc:1428] Setting key 3b31382d1a95d74a81e4aed8433a6acc
I20260806 02:51:35.933380 25127 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41437 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:51:35 dist-test-slave-8tdl krb5kdc[24969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984695, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.125@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:51:35.956014 25129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:35.936224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47810 (local address 127.1.210.126:41437)
0806 02:51:35.936817 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:35.936825 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:35.937835 (+ 1010us) server_negotiation.cc:407] Connection header received
0806 02:51:35.939927 (+ 2092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:35.939935 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:35.940036 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:35.940183 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:35.941971 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:35.943280 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:35.944922 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:35.945662 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:35.945808 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:35.949744 (+ 3936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:35.949750 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:35.949753 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:35.949788 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:35.952734 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:35.953551 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:35.953560 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:35.953564 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:35.953696 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:35.954148 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:35.954153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:35.954156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:35.954453 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:35.954629 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:35.955396 (+ 767us) server_negotiation.cc:299] Negotiation successful
0806 02:51:35.955628 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":294,"threads_started":1}
I20260806 02:51:35.957015 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:35.936790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41437 (local address 127.0.0.1:47810)
0806 02:51:35.937636 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:35.937684 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:35.939628 (+ 1944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:35.940446 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:35.940456 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:35.941126 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:35.941712 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:35.941730 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:35.943470 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:35.943474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:35.944751 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:35.944772 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:35.945953 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:35.945960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:35.946186 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:35.946953 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:35.946976 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:35.949484 (+ 2508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:35.952911 (+ 3427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:35.952921 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:35.952942 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:35.953420 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:35.953813 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:35.953818 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:35.953822 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:35.954020 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:35.954596 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:35.954605 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:35.955035 (+ 430us) client_negotiation.cc:769] Sending connection context
0806 02:51:35.955318 (+ 283us) client_negotiation.cc:240] Negotiation successful
0806 02:51:35.955867 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":362,"threads_started":1}
I20260806 02:51:35.959002 25127 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.125" port: 45019 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.210.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.210.124@KRBTEST.COM" created.
Aug 06 02:51:35 dist-test-slave-8tdl krb5kdc[24969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984695, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.125@KRBTEST.COM for kudu/127.1.210.125@KRBTEST.COM
I20260806 02:51:35.974854 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:35.959964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:45019 (local address 127.0.0.1:36654)
0806 02:51:35.960183 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:35.960199 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:35.960292 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:35.961254 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:35.961258 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:35.961703 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:35.961835 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:35.961844 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:35.963346 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:35.963351 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:35.964224 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:35.964235 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:35.965078 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:35.965083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:35.965285 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:35.965823 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:35.965840 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:35.968979 (+ 3139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:35.972314 (+ 3335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:35.972320 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:35.972324 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:35.972680 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:35.973081 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:35.973085 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:35.973089 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:35.973209 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:35.973828 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:35.973833 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:35.973974 (+ 141us) client_negotiation.cc:769] Sending connection context
0806 02:51:35.974123 (+ 149us) client_negotiation.cc:240] Negotiation successful
0806 02:51:35.974416 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 02:51:35.974975 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:35.960194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36654 (local address 127.1.210.125:45019)
0806 02:51:35.960839 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:35.960845 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:35.960869 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:51:35.960912 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:35.960917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:35.961109 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:35.961263 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:35.962020 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:35.963211 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:35.964361 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:35.964921 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:35.965057 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:35.969203 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:35.969214 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:35.969229 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:35.969258 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:35.972178 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:35.972813 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:35.972828 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:35.972830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:35.972943 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:35.973311 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:35.973319 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:35.973322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:35.973679 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:35.973908 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:35.974260 (+ 352us) server_negotiation.cc:299] Negotiation successful
0806 02:51:35.974537 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":494,"thread_start_us":367,"threads_started":1}
I20260806 02:51:35.978323 25127 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.124" port: 41239 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:51:35.979768 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:35.979232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:41239 (local address 127.0.0.1:43054)
0806 02:51:35.979427 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:35.979586 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260806 02:51:35.980068 25076 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.210.124:41239: connect: Connection refused (error 111)
I20260806 02:51:35.981828 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:35.981475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:41239 (local address 127.0.0.1:43070)
0806 02:51:35.981632 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:35.981712 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 02:51:35.982076 25127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.124:41239: Network error: Client connection negotiation failed: client connection to 127.1.210.124:41239: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.210.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.124.keytab.
Entry for principal kudu/127.1.210.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.124.keytab.
I20260806 02:51:36.011638 1865 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
/tmp/dist-test-task5Fb5fs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.124:41239
--webserver_interface=127.1.210.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.124.keytab
--principal=kudu/127.1.210.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.210.84:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.210.126:41437,127.1.210.125:45019,127.1.210.124:41239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:51:36.031421 25127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.124" port: 41239 } attempt: 1
I20260806 02:51:36.032776 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:36.032284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:41239 (local address 127.0.0.1:43086)
0806 02:51:36.032503 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:36.032618 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260806 02:51:36.033811 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:36.033450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:41239 (local address 127.0.0.1:43100)
0806 02:51:36.033603 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:36.033702 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 02:51:36.034029 25127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.124:41239: Network error: Client connection negotiation failed: client connection to 127.1.210.124:41239: connect: Connection refused (error 111)
I20260806 02:51:36.076205 25051 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.125" port: 45019 } attempt: 5
Aug 06 02:51:36 dist-test-slave-8tdl krb5kdc[24969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984695, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for kudu/127.1.210.125@KRBTEST.COM
I20260806 02:51:36.090801 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:36.077016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:45019 (local address 127.0.0.1:36658)
0806 02:51:36.077336 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:36.077366 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:36.077561 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:36.077976 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:36.077984 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:36.078669 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:36.079042 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:36.079061 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:36.080137 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:36.080144 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:36.081100 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:36.081114 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:36.081887 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:36.081893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:36.082131 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:36.082831 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:36.082850 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:36.084953 (+ 2103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:36.088179 (+ 3226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:36.088185 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:36.088189 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:36.088601 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:36.088989 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:36.088995 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:36.088998 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:36.089157 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:36.089725 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:36.089730 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:36.089901 (+ 171us) client_negotiation.cc:769] Sending connection context
0806 02:51:36.090059 (+ 158us) client_negotiation.cc:240] Negotiation successful
0806 02:51:36.090395 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260806 02:51:36.090801 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:36.077213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36658 (local address 127.1.210.125:45019)
0806 02:51:36.077433 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:36.077442 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:36.077467 (+ 25us) server_negotiation.cc:407] Connection header received
0806 02:51:36.077718 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:36.077721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:36.077787 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:36.077873 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:36.079248 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:36.079972 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:36.081277 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:36.081726 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:36.081869 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:36.085170 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:36.085175 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:36.085178 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:36.085212 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:36.088013 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:36.088726 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:36.088732 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:36.088736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:36.088872 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:36.089273 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:36.089279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:36.089282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:36.089600 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:36.089761 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:36.090167 (+ 406us) server_negotiation.cc:299] Negotiation successful
0806 02:51:36.090399 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 02:51:36.092010 25051 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.124" port: 41239 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:51:36.093196 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:36.092746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:41239 (local address 127.0.0.1:43106)
0806 02:51:36.092938 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:36.093042 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260806 02:51:36.094250 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:36.093940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:41239 (local address 127.0.0.1:43108)
0806 02:51:36.094073 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:36.094154 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 02:51:36.094540 25051 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.124:41239: Network error: Client connection negotiation failed: client connection to 127.1.210.124:41239: connect: Connection refused (error 111)
I20260806 02:51:36.102289 25127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.124" port: 41239 } attempt: 2
I20260806 02:51:36.103456 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:36.103001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:41239 (local address 127.0.0.1:43110)
0806 02:51:36.103206 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:36.103302 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260806 02:51:36.104529 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:36.104183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:41239 (local address 127.0.0.1:43126)
0806 02:51:36.104335 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:36.104411 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 02:51:36.104766 25127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.124:41239: Network error: Client connection negotiation failed: client connection to 127.1.210.124:41239: connect: Connection refused (error 111)
I20260806 02:51:36.145803 25051 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.124" port: 41239 } attempt: 1
I20260806 02:51:36.147210 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:36.146658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:41239 (local address 127.0.0.1:43128)
0806 02:51:36.146927 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:36.147034 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260806 02:51:36.148264 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:36.147913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:41239 (local address 127.0.0.1:43138)
0806 02:51:36.148072 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:36.148157 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260806 02:51:36.148485 25051 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.124:41239: Network error: Client connection negotiation failed: client connection to 127.1.210.124:41239: connect: Connection refused (error 111)
I20260806 02:51:36.196019 25127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.124" port: 41239 } attempt: 3
I20260806 02:51:36.197254 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:36.196756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:41239 (local address 127.0.0.1:43140)
0806 02:51:36.196992 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:36.197090 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260806 02:51:36.198338 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:36.197876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:41239 (local address 127.0.0.1:43148)
0806 02:51:36.198139 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:36.198218 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260806 02:51:36.198547 25127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.124:41239: Network error: Client connection negotiation failed: client connection to 127.1.210.124:41239: connect: Connection refused (error 111)
I20260806 02:51:36.216682 25051 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.124" port: 41239 } attempt: 2
I20260806 02:51:36.217885 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:36.217367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:41239 (local address 127.0.0.1:43150)
0806 02:51:36.217616 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:36.217705 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260806 02:51:36.218918 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:36.218533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:41239 (local address 127.0.0.1:43166)
0806 02:51:36.218744 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:36.218814 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 02:51:36.219122 25051 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.124:41239: Network error: Client connection negotiation failed: client connection to 127.1.210.124:41239: connect: Connection refused (error 111)
I20260806 02:51:36.325356 25051 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.124" port: 41239 } attempt: 3
I20260806 02:51:36.326551 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:36.326064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:41239 (local address 127.0.0.1:43174)
0806 02:51:36.326277 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:36.326377 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260806 02:51:36.327637 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:36.327238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:41239 (local address 127.0.0.1:43182)
0806 02:51:36.327469 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:36.327539 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260806 02:51:36.327827 25051 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.124:41239: Network error: Client connection negotiation failed: client connection to 127.1.210.124:41239: connect: Connection refused (error 111)
I20260806 02:51:36.341780 25127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.124" port: 41239 } attempt: 4
I20260806 02:51:36.343032 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:36.342517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:41239 (local address 127.0.0.1:43198)
0806 02:51:36.342789 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:36.342875 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260806 02:51:36.344063 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:36.343691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:41239 (local address 127.0.0.1:43204)
0806 02:51:36.343874 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:36.343958 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260806 02:51:36.344273 25127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.124:41239: Network error: Client connection negotiation failed: client connection to 127.1.210.124:41239: connect: Connection refused (error 111)
W20260806 02:51:36.385222 25135 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 02:51:36.385614 25135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:36.385735 25135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:36.385869 25135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:36.393304 25135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:51:36.393431 25135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:36.393566 25135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:36.393653 25135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:51:36.393759 25135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:51:36.403712 25135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.210.84:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.210.126:41437,127.1.210.125:45019,127.1.210.124:41239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/krb5kdc/kudu_127.1.210.124.keytab
--principal=kudu/127.1.210.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.210.124:41239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.210.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:51:36.405076 25135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:36.406451 25135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:36.418460 25143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:36.418489 25141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:36.418440 25140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:36.419703 25135 server_base.cc:1034] running on GCE node
I20260806 02:51:36.420477 25135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:36.421947 25135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:36.423207 25135 hybrid_clock.cc:648] HybridClock initialized: now 1785984696423142 us; error 37 us; skew 500 ppm
I20260806 02:51:36.423561 25135 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:51:36 dist-test-slave-8tdl krb5kdc[24969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984696, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:36.426765 25135 init.cc:377] Logged in from keytab as kudu/127.1.210.124@KRBTEST.COM (short username kudu)
I20260806 02:51:36.428320 25135 webserver.cc:460] Webserver started at http://127.1.210.124:42021/ using document root <none> and password file <none>
I20260806 02:51:36.429041 25135 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:36.429181 25135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:36.429512 25135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:36.432824 25135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-2/data/instance:
uuid: "7d6da2a87e014aa3891c9a8d3b65a2ab"
format_stamp: "Formatted at 2026-08-06 02:51:36 on dist-test-slave-8tdl"
server_key: "4c0c7b5f18c7772df693afe9a7241e77"
server_key_iv: "e9176c2de28f4063cddfd19c1de64959"
server_key_version: "encryptionkey@0"
I20260806 02:51:36.433641 25135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-2/wal/instance:
uuid: "7d6da2a87e014aa3891c9a8d3b65a2ab"
format_stamp: "Formatted at 2026-08-06 02:51:36 on dist-test-slave-8tdl"
server_key: "4c0c7b5f18c7772df693afe9a7241e77"
server_key_iv: "e9176c2de28f4063cddfd19c1de64959"
server_key_version: "encryptionkey@0"
I20260806 02:51:36.439255 25135 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260806 02:51:36.443027 25150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:36.444603 25135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:51:36.444799 25135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-2/data,/tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-2/wal
uuid: "7d6da2a87e014aa3891c9a8d3b65a2ab"
format_stamp: "Formatted at 2026-08-06 02:51:36 on dist-test-slave-8tdl"
server_key: "4c0c7b5f18c7772df693afe9a7241e77"
server_key_iv: "e9176c2de28f4063cddfd19c1de64959"
server_key_version: "encryptionkey@0"
I20260806 02:51:36.445036 25135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:36.485595 25135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:36.490199 25135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:36.490528 25135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:36.505048 25051 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.124" port: 41239 } attempt: 4
I20260806 02:51:36.506474 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:36.505840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:41239 (local address 127.0.0.1:43220)
0806 02:51:36.506120 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:36.506247 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260806 02:51:36.507786 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:36.507288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:41239 (local address 127.0.0.1:43234)
0806 02:51:36.507498 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:36.507607 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.210.124:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260806 02:51:36.508092 25051 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.210.124:41239: Network error: Client connection negotiation failed: client connection to 127.1.210.124:41239: connect: Connection refused (error 111)
I20260806 02:51:36.510831 25135 rpc_server.cc:307] RPC server started. Bound to: 127.1.210.124:41239
I20260806 02:51:36.510852 25201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.210.124:41239 every 8 connection(s)
I20260806 02:51:36.512284 25135 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-2/data/info.pb
I20260806 02:51:36.516291 25202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:36.519070 1865 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu as pid 25135
I20260806 02:51:36.519270 1865 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5Fb5fs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984304136196-1865-0/minicluster-data/master-2/wal/instance
I20260806 02:51:36.519814 1865 external_mini_cluster.cc:1428] Setting key 6626517532ed5d07dcb985c38d0e345d
I20260806 02:51:36.525394 25202 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41437 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:51:36 dist-test-slave-8tdl krb5kdc[24969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.124@KRBTEST.COM
Aug 06 02:51:36 dist-test-slave-8tdl krb5kdc[24969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984696, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.124@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:51:36.545224 25207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:36.528346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47812 (local address 127.1.210.126:41437)
0806 02:51:36.529687 (+ 1341us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:36.529692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:36.529708 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:51:36.529758 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:36.529761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:36.529819 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:36.529951 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:36.531272 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:36.531971 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:36.533495 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:36.533919 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:36.534049 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:36.538349 (+ 4300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:36.538359 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:36.538365 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:36.538421 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:36.542139 (+ 3718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:36.543005 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:36.543011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:36.543015 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:36.543159 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:36.543545 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:36.543550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:36.543555 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:36.543824 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:36.543986 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:36.544567 (+ 581us) server_negotiation.cc:299] Negotiation successful
0806 02:51:36.544793 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1240,"thread_start_us":1126,"threads_started":1}
I20260806 02:51:36.546574 25205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:36.521768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43250 (local address 127.1.210.124:41239)
0806 02:51:36.522596 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:36.522613 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:36.522692 (+ 79us) server_negotiation.cc:407] Connection header received
0806 02:51:36.525089 (+ 2397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:36.525117 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:36.525672 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:36.526139 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:36.527780 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:36.529872 (+ 2092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:36.531509 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:36.532188 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:36.532434 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:36.537016 (+ 4582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:36.537035 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:36.537057 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:36.537097 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:36.541597 (+ 4500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:36.542495 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:36.542514 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:36.542518 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:36.542775 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:36.543229 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:36.543234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:36.543237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:36.543830 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:36.544190 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:36.544518 (+ 328us) server_negotiation.cc:299] Negotiation successful
0806 02:51:36.544933 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":566,"thread_start_us":357,"threads_started":1}
I20260806 02:51:36.546605 25206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:36.528284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.126:41437 (local address 127.0.0.1:47812)
0806 02:51:36.529094 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:36.529144 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:36.529276 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:36.530020 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:36.530027 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:36.530743 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:36.531100 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:36.531111 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:36.532147 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:36.532151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:36.533342 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:36.533355 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:36.534188 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:36.534197 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:36.534349 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:36.535114 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:36.535138 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:36.537957 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:36.542300 (+ 4343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:36.542315 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:36.542335 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:36.542854 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:36.543280 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:36.543285 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:36.543289 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:36.543426 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:36.543969 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:36.543977 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:36.544197 (+ 220us) client_negotiation.cc:769] Sending connection context
0806 02:51:36.544448 (+ 251us) client_negotiation.cc:240] Negotiation successful
0806 02:51:36.545112 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":222,"thread_start_us":398,"threads_started":1}
I20260806 02:51:36.549144 25202 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.125" port: 45019 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:51:36 dist-test-slave-8tdl krb5kdc[24969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984696, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.124@KRBTEST.COM for kudu/127.1.210.125@KRBTEST.COM
Aug 06 02:51:36 dist-test-slave-8tdl krb5kdc[24969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.126@KRBTEST.COM
I20260806 02:51:36.566069 25206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:36.550066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.125:45019 (local address 127.0.0.1:36666)
0806 02:51:36.550327 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:36.550349 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:36.550484 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:36.551268 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:36.551273 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:36.551796 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:36.552021 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:36.552037 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:36.553384 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:36.553389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:36.554932 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:36.554949 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:36.555791 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:36.555796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:36.555977 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:36.556658 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:36.556685 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:36.559367 (+ 2682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:36.563136 (+ 3769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:36.563144 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:36.563152 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:36.563762 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:36.564230 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:36.564237 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:36.564241 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:36.564443 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:36.565019 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:36.565024 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:36.565222 (+ 198us) client_negotiation.cc:769] Sending connection context
0806 02:51:36.565388 (+ 166us) client_negotiation.cc:240] Negotiation successful
0806 02:51:36.565645 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260806 02:51:36.566118 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:36.550393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36666 (local address 127.1.210.125:45019)
0806 02:51:36.550939 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:36.550946 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:36.550966 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:51:36.551027 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:36.551032 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:36.551104 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:36.551203 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:36.552209 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:36.553228 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:36.555131 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:36.555619 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:36.555758 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:36.559660 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:36.559671 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:36.559674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:36.559714 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:36.562970 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:36.563913 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:36.563918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:36.563920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:36.564069 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:36.564561 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:36.564565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:36.564567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:36.564879 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:36.564999 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:36.565525 (+ 526us) server_negotiation.cc:299] Negotiation successful
0806 02:51:36.565806 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346}
I20260806 02:51:36.567334 25202 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.210.124" port: 41239 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:51:36.567929 25207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:36.550605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47820 (local address 127.1.210.126:41437)
0806 02:51:36.550876 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:36.550882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:36.550903 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:51:36.550990 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:36.550994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:36.551088 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:36.551197 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:36.552687 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:36.553632 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:36.555564 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:36.556138 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:36.556295 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:36.561082 (+ 4787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:36.561087 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:36.561091 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:36.561127 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:36.564548 (+ 3421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:36.565452 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:36.565458 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:36.565461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:36.565625 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:36.566110 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:36.566117 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:36.566120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:36.566648 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:36.566995 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:36.567368 (+ 373us) server_negotiation.cc:299] Negotiation successful
0806 02:51:36.567617 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
Aug 06 02:51:36 dist-test-slave-8tdl krb5kdc[24969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984696, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.124@KRBTEST.COM for kudu/127.1.210.124@KRBTEST.COM
Aug 06 02:51:36 dist-test-slave-8tdl krb5kdc[24969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.210.125@KRBTEST.COM
I20260806 02:51:36.582182 25205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:36.568322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43260 (local address 127.1.210.124:41239)
0806 02:51:36.568522 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:36.568528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:36.568709 (+ 181us) server_negotiation.cc:407] Connection header received
0806 02:51:36.568813 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:36.568820 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:36.568902 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:36.569023 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:36.569994 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:36.571113 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:36.572335 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:36.572758 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:36.572899 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:36.576131 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:36.576181 (+ 50us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:36.576186 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:36.576235 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:36.579595 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:36.580255 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:36.580263 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:36.580266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:36.580395 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:36.580753 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:36.580761 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:36.580764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:36.581061 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:36.581218 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:36.581559 (+ 341us) server_negotiation.cc:299] Negotiation successful
0806 02:51:36.581784 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 02:51:36.582293 25206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:36.568321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:41239 (local address 127.0.0.1:43260)
0806 02:51:36.568622 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:36.568635 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:36.568724 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:36.569075 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:36.569081 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:36.569611 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:36.569820 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:36.569839 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:36.571273 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:36.571277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:36.572182 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:36.572194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:36.572907 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:36.572912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:36.573132 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:36.573748 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:36.573767 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:36.575941 (+ 2174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:36.579736 (+ 3795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:36.579744 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:36.579748 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:36.580135 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:36.580504 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:36.580512 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:36.580516 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:36.580651 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:36.581180 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:36.581185 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:36.581338 (+ 153us) client_negotiation.cc:769] Sending connection context
0806 02:51:36.581485 (+ 147us) client_negotiation.cc:240] Negotiation successful
0806 02:51:36.581933 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":141}
I20260806 02:51:36.585232 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:36.569921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36676 (local address 127.1.210.125:45019)
0806 02:51:36.570190 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:36.570194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:36.570209 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:51:36.570250 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:36.570254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:36.570309 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:36.570385 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:36.571901 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:36.572608 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:36.574016 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:36.574454 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:36.574598 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:36.579489 (+ 4891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:36.579495 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:36.579498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:36.579536 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:36.582193 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:36.583100 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:36.583104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:36.583106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:36.583247 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:36.583701 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:36.583706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:36.583710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:36.584078 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:36.584356 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:36.584658 (+ 302us) server_negotiation.cc:299] Negotiation successful
0806 02:51:36.584899 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 02:51:36.588508 25202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d6da2a87e014aa3891c9a8d3b65a2ab: Bootstrap starting.
I20260806 02:51:36.590834 1865 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:51:36.592458 25202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d6da2a87e014aa3891c9a8d3b65a2ab: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:36.593918 25202 log.cc:824] T 00000000000000000000000000000000 P 7d6da2a87e014aa3891c9a8d3b65a2ab: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:36.597195 25202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d6da2a87e014aa3891c9a8d3b65a2ab: No bootstrap required, opened a new log
I20260806 02:51:36.594136 25025 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "55c0f81eb98548ff80d7374a3843aa91"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47820
I20260806 02:51:36.604559 25202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d6da2a87e014aa3891c9a8d3b65a2ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c0f81eb98548ff80d7374a3843aa91" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41437 } } peers { permanent_uuid: "c44a3437a4fe44c5beb39ad2d0ccdd1f" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 45019 } } peers { permanent_uuid: "7d6da2a87e014aa3891c9a8d3b65a2ab" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 41239 } }
I20260806 02:51:36.604944 25202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d6da2a87e014aa3891c9a8d3b65a2ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d6da2a87e014aa3891c9a8d3b65a2ab, State: Initialized, Role: FOLLOWER
I20260806 02:51:36.605412 25202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d6da2a87e014aa3891c9a8d3b65a2ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c0f81eb98548ff80d7374a3843aa91" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41437 } } peers { permanent_uuid: "c44a3437a4fe44c5beb39ad2d0ccdd1f" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 45019 } } peers { permanent_uuid: "7d6da2a87e014aa3891c9a8d3b65a2ab" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 41239 } }
I20260806 02:51:36.606424 25209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d6da2a87e014aa3891c9a8d3b65a2ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c0f81eb98548ff80d7374a3843aa91" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41437 } } peers { permanent_uuid: "c44a3437a4fe44c5beb39ad2d0ccdd1f" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 45019 } } peers { permanent_uuid: "7d6da2a87e014aa3891c9a8d3b65a2ab" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 41239 } } }
I20260806 02:51:36.606901 25209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d6da2a87e014aa3891c9a8d3b65a2ab [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:51:36.607577 25202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d6da2a87e014aa3891c9a8d3b65a2ab [sys.catalog]: configured and running, proceeding with master startup.
W20260806 02:51:36.617321 25220 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7d6da2a87e014aa3891c9a8d3b65a2ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:51:36.617471 25220 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:51:36.615705 25135 master_runner.cc:186] Error getting master registration for 127.1.210.126:41437: OK, instance_id { permanent_uuid: "55c0f81eb98548ff80d7374a3843aa91" instance_seqno: 1785984695356005 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 02:51:36.627345 25135 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 02:51:36.643493 25127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.124" port: 41239 } attempt: 5
Aug 06 02:51:36 dist-test-slave-8tdl krb5kdc[24969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984695, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.125@KRBTEST.COM for kudu/127.1.210.124@KRBTEST.COM
I20260806 02:51:36.661659 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:36.644297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:41239 (local address 127.0.0.1:43274)
0806 02:51:36.644527 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:36.644549 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:36.644684 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:36.645074 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:36.645081 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:36.645619 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:36.645841 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:36.645854 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:36.647436 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:36.647444 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:36.649365 (+ 1921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:36.649388 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:36.650615 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:36.650622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:36.650972 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:36.651842 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:36.651869 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:36.654531 (+ 2662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:36.658533 (+ 4002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:36.658540 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:36.658544 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:36.659157 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:36.659611 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:36.659616 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:36.659620 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:36.659801 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:36.660567 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:36.660572 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:36.660741 (+ 169us) client_negotiation.cc:769] Sending connection context
0806 02:51:36.660911 (+ 170us) client_negotiation.cc:240] Negotiation successful
0806 02:51:36.661244 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260806 02:51:36.661870 25205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:36.644418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43274 (local address 127.1.210.124:41239)
0806 02:51:36.644657 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:36.644665 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:36.644695 (+ 30us) server_negotiation.cc:407] Connection header received
0806 02:51:36.644806 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:36.644811 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:36.644897 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:36.645030 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:36.646023 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:36.647270 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:36.649585 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:36.650392 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:36.650539 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:36.654842 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:36.654849 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:36.654889 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:36.654940 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:36.658378 (+ 3438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:36.659296 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:36.659305 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:36.659310 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:36.659474 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:36.659923 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:36.659928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:36.659932 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:36.660422 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:36.660620 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:36.661032 (+ 412us) server_negotiation.cc:299] Negotiation successful
0806 02:51:36.661302 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 02:51:36.666990 25127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c44a3437a4fe44c5beb39ad2d0ccdd1f: Bootstrap starting.
I20260806 02:51:36.671473 25127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c44a3437a4fe44c5beb39ad2d0ccdd1f: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:36.672909 25127 log.cc:824] T 00000000000000000000000000000000 P c44a3437a4fe44c5beb39ad2d0ccdd1f: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:36.676519 25127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c44a3437a4fe44c5beb39ad2d0ccdd1f: No bootstrap required, opened a new log
I20260806 02:51:36.689011 25127 raft_consensus.cc:348] T 00000000000000000000000000000000 P c44a3437a4fe44c5beb39ad2d0ccdd1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c0f81eb98548ff80d7374a3843aa91" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41437 } } peers { permanent_uuid: "c44a3437a4fe44c5beb39ad2d0ccdd1f" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 45019 } } peers { permanent_uuid: "7d6da2a87e014aa3891c9a8d3b65a2ab" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 41239 } }
I20260806 02:51:36.689468 25127 raft_consensus.cc:729] T 00000000000000000000000000000000 P c44a3437a4fe44c5beb39ad2d0ccdd1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c44a3437a4fe44c5beb39ad2d0ccdd1f, State: Initialized, Role: FOLLOWER
I20260806 02:51:36.690012 25127 consensus_queue.cc:260] T 00000000000000000000000000000000 P c44a3437a4fe44c5beb39ad2d0ccdd1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c0f81eb98548ff80d7374a3843aa91" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41437 } } peers { permanent_uuid: "c44a3437a4fe44c5beb39ad2d0ccdd1f" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 45019 } } peers { permanent_uuid: "7d6da2a87e014aa3891c9a8d3b65a2ab" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 41239 } }
I20260806 02:51:36.691181 25223 sys_catalog.cc:455] T 00000000000000000000000000000000 P c44a3437a4fe44c5beb39ad2d0ccdd1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c0f81eb98548ff80d7374a3843aa91" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41437 } } peers { permanent_uuid: "c44a3437a4fe44c5beb39ad2d0ccdd1f" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 45019 } } peers { permanent_uuid: "7d6da2a87e014aa3891c9a8d3b65a2ab" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 41239 } } }
I20260806 02:51:36.691690 25223 sys_catalog.cc:458] T 00000000000000000000000000000000 P c44a3437a4fe44c5beb39ad2d0ccdd1f [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:51:36.692510 25127 sys_catalog.cc:564] T 00000000000000000000000000000000 P c44a3437a4fe44c5beb39ad2d0ccdd1f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:51:36.700543 25058 master_runner.cc:186] Error getting master registration for 127.1.210.126:41437: OK, instance_id { permanent_uuid: "55c0f81eb98548ff80d7374a3843aa91" instance_seqno: 1785984695356005 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 02:51:36.711603 25058 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260806 02:51:36.714630 25234 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c44a3437a4fe44c5beb39ad2d0ccdd1f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:51:36.714809 25234 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260806 02:51:36.742991 25156 tablet.cc:2369] T 00000000000000000000000000000000 P 7d6da2a87e014aa3891c9a8d3b65a2ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:51:36.785292 25051 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.210.124" port: 41239 } attempt: 5
Aug 06 02:51:36 dist-test-slave-8tdl krb5kdc[24969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984695, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.210.126@KRBTEST.COM for kudu/127.1.210.124@KRBTEST.COM
I20260806 02:51:36.803887 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:36.786157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.210.124:41239 (local address 127.0.0.1:43284)
0806 02:51:36.786470 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:36.786492 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:36.786621 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:36.787076 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:36.787081 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:36.787630 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:36.787856 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:36.787869 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:36.789396 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:36.789402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:36.790988 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:36.791005 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:36.792050 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:36.792058 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:36.792270 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:36.793087 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:36.793113 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:36.796339 (+ 3226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:36.800871 (+ 4532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:36.800877 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:36.800881 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:36.801435 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:36.801905 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:36.801911 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:36.801915 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:36.802131 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:36.802781 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:36.802786 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:36.802961 (+ 175us) client_negotiation.cc:769] Sending connection context
0806 02:51:36.803129 (+ 168us) client_negotiation.cc:240] Negotiation successful
0806 02:51:36.803437 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177}
I20260806 02:51:36.803918 25205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:36.786314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43284 (local address 127.1.210.124:41239)
0806 02:51:36.786565 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:36.786572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:36.786596 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:51:36.786791 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:36.786797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:36.786882 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:36.787010 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:36.788069 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:36.789225 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:36.791137 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:36.791865 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:36.792004 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:36.796558 (+ 4554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:36.796565 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:36.796570 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:36.796612 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:36.800689 (+ 4077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:36.801592 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:36.801598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:36.801602 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:36.801777 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:36.802275 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:36.802281 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:36.802285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:36.802636 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:36.802811 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:36.803280 (+ 469us) server_negotiation.cc:299] Negotiation successful
0806 02:51:36.803530 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260806 02:51:36.811174 25051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55c0f81eb98548ff80d7374a3843aa91: Bootstrap starting.
I20260806 02:51:36.815642 25051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55c0f81eb98548ff80d7374a3843aa91: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:36.817152 25051 log.cc:824] T 00000000000000000000000000000000 P 55c0f81eb98548ff80d7374a3843aa91: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:36.827045 25051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55c0f81eb98548ff80d7374a3843aa91: No bootstrap required, opened a new log
I20260806 02:51:36.835414 25051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55c0f81eb98548ff80d7374a3843aa91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c0f81eb98548ff80d7374a3843aa91" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41437 } } peers { permanent_uuid: "c44a3437a4fe44c5beb39ad2d0ccdd1f" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 45019 } } peers { permanent_uuid: "7d6da2a87e014aa3891c9a8d3b65a2ab" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 41239 } }
I20260806 02:51:36.835968 25051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55c0f81eb98548ff80d7374a3843aa91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55c0f81eb98548ff80d7374a3843aa91, State: Initialized, Role: FOLLOWER
I20260806 02:51:36.836583 25051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55c0f81eb98548ff80d7374a3843aa91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c0f81eb98548ff80d7374a3843aa91" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41437 } } peers { permanent_uuid: "c44a3437a4fe44c5beb39ad2d0ccdd1f" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 45019 } } peers { permanent_uuid: "7d6da2a87e014aa3891c9a8d3b65a2ab" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 41239 } }
I20260806 02:51:36.838019 25237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55c0f81eb98548ff80d7374a3843aa91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c0f81eb98548ff80d7374a3843aa91" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41437 } } peers { permanent_uuid: "c44a3437a4fe44c5beb39ad2d0ccdd1f" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 45019 } } peers { permanent_uuid: "7d6da2a87e014aa3891c9a8d3b65a2ab" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 41239 } } }
I20260806 02:51:36.838744 25237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55c0f81eb98548ff80d7374a3843aa91 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:51:36.839197 25051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55c0f81eb98548ff80d7374a3843aa91 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:51:36.839267 25025 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55c0f81eb98548ff80d7374a3843aa91 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 02:51:36.839469 25025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55c0f81eb98548ff80d7374a3843aa91 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:36.843389 25025 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55c0f81eb98548ff80d7374a3843aa91 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c0f81eb98548ff80d7374a3843aa91" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41437 } } peers { permanent_uuid: "c44a3437a4fe44c5beb39ad2d0ccdd1f" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 45019 } } peers { permanent_uuid: "7d6da2a87e014aa3891c9a8d3b65a2ab" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 41239 } }
W20260806 02:51:36.844332 25004 tablet.cc:2369] T 00000000000000000000000000000000 P 55c0f81eb98548ff80d7374a3843aa91: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:51:36.846791 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "55c0f81eb98548ff80d7374a3843aa91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c44a3437a4fe44c5beb39ad2d0ccdd1f"
I20260806 02:51:36.847302 25100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c44a3437a4fe44c5beb39ad2d0ccdd1f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:36.850996 25100 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c44a3437a4fe44c5beb39ad2d0ccdd1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55c0f81eb98548ff80d7374a3843aa91 in term 1.
I20260806 02:51:36.851725 25002 leader_election.cc:302] T 00000000000000000000000000000000 P 55c0f81eb98548ff80d7374a3843aa91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55c0f81eb98548ff80d7374a3843aa91, c44a3437a4fe44c5beb39ad2d0ccdd1f; no voters:
I20260806 02:51:36.853206 25177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "55c0f81eb98548ff80d7374a3843aa91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7d6da2a87e014aa3891c9a8d3b65a2ab"
I20260806 02:51:36.853669 25177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d6da2a87e014aa3891c9a8d3b65a2ab [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:36.857143 25177 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7d6da2a87e014aa3891c9a8d3b65a2ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55c0f81eb98548ff80d7374a3843aa91 in term 1.
I20260806 02:51:36.857931 25025 leader_election.cc:288] T 00000000000000000000000000000000 P 55c0f81eb98548ff80d7374a3843aa91 [CANDIDATE]: Term 1 election: Requested vote from peers c44a3437a4fe44c5beb39ad2d0ccdd1f (127.1.210.125:45019), 7d6da2a87e014aa3891c9a8d3b65a2ab (127.1.210.124:41239)
I20260806 02:51:36.864960 25237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55c0f81eb98548ff80d7374a3843aa91 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:51:36.875238 25237 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55c0f81eb98548ff80d7374a3843aa91 [term 1 LEADER]: Becoming Leader. State: Replica: 55c0f81eb98548ff80d7374a3843aa91, State: Running, Role: LEADER
I20260806 02:51:36.877120 25237 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55c0f81eb98548ff80d7374a3843aa91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c0f81eb98548ff80d7374a3843aa91" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41437 } } peers { permanent_uuid: "c44a3437a4fe44c5beb39ad2d0ccdd1f" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 45019 } } peers { permanent_uuid: "7d6da2a87e014aa3891c9a8d3b65a2ab" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 41239 } }
I20260806 02:51:36.887387 25243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55c0f81eb98548ff80d7374a3843aa91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55c0f81eb98548ff80d7374a3843aa91. Latest consensus state: current_term: 1 leader_uuid: "55c0f81eb98548ff80d7374a3843aa91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c0f81eb98548ff80d7374a3843aa91" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41437 } } peers { permanent_uuid: "c44a3437a4fe44c5beb39ad2d0ccdd1f" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 45019 } } peers { permanent_uuid: "7d6da2a87e014aa3891c9a8d3b65a2ab" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 41239 } } }
I20260806 02:51:36.887986 25243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55c0f81eb98548ff80d7374a3843aa91 [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:36.893496 25251 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260806 02:51:36.903494 25079 tablet.cc:2369] T 00000000000000000000000000000000 P c44a3437a4fe44c5beb39ad2d0ccdd1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:51:36.906060 25251 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:51:36.913341 25257 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 02:51:36.952358 25205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:36.911814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43294 (local address 127.1.210.124:41239)
0806 02:51:36.912030 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:36.912037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:36.920515 (+ 8478us) server_negotiation.cc:407] Connection header received
0806 02:51:36.920632 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:36.920637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:36.920746 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:36.920866 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:36.922832 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:36.923917 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:36.929955 (+ 6038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:36.930637 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:36.932804 (+ 2167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:36.942550 (+ 9746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:36.942558 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:36.942564 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:36.942614 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:36.947065 (+ 4451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:36.949000 (+ 1935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:36.949009 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:36.949013 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:36.949197 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:36.949682 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:36.949688 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:36.949692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:36.950197 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:36.950466 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:36.951567 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0806 02:51:36.951879 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 02:51:36.957509 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:36.917412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36682 (local address 127.1.210.125:45019)
0806 02:51:36.917602 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:36.917609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:36.927290 (+ 9681us) server_negotiation.cc:407] Connection header received
0806 02:51:36.931190 (+ 3900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:36.931200 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:36.931289 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:36.931359 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:36.933701 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:36.934951 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:36.939369 (+ 4418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:36.940113 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:36.940221 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:36.947942 (+ 7721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:36.947950 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:36.947955 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:36.948006 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:36.952119 (+ 4113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:36.953888 (+ 1769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:36.953896 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:36.953914 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:36.954118 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:36.955306 (+ 1188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:36.955311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:36.955315 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:36.955794 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:36.956089 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:36.956771 (+ 682us) server_negotiation.cc:299] Negotiation successful
0806 02:51:36.957075 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 02:51:36.968461 25207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:36.908780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47836 (local address 127.1.210.126:41437)
0806 02:51:36.908988 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:36.908995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:36.918744 (+ 9749us) server_negotiation.cc:407] Connection header received
0806 02:51:36.919069 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:36.919076 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:36.919163 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:36.919240 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:36.926838 (+ 7598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:36.927980 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:36.936400 (+ 8420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:36.937077 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:36.937175 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:36.951166 (+ 13991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:36.951175 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:36.951180 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:36.951236 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:36.963640 (+ 12404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:36.964693 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:36.964700 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:36.964704 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:36.964879 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:36.965441 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:36.965447 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:36.965450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:36.965911 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:36.966247 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:36.967682 (+ 1435us) server_negotiation.cc:299] Negotiation successful
0806 02:51:36.967987 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 02:51:36.974792 25100 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c44a3437a4fe44c5beb39ad2d0ccdd1f [term 1 FOLLOWER]: Refusing update from remote peer 55c0f81eb98548ff80d7374a3843aa91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:51:36.974797 25177 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7d6da2a87e014aa3891c9a8d3b65a2ab [term 1 FOLLOWER]: Refusing update from remote peer 55c0f81eb98548ff80d7374a3843aa91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:51:36.975925 25237 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 55c0f81eb98548ff80d7374a3843aa91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c44a3437a4fe44c5beb39ad2d0ccdd1f" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 45019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:51:36.977372 25237 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 55c0f81eb98548ff80d7374a3843aa91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d6da2a87e014aa3891c9a8d3b65a2ab" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 41239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:51:37.001699 25223 sys_catalog.cc:455] T 00000000000000000000000000000000 P c44a3437a4fe44c5beb39ad2d0ccdd1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55c0f81eb98548ff80d7374a3843aa91. Latest consensus state: current_term: 1 leader_uuid: "55c0f81eb98548ff80d7374a3843aa91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c0f81eb98548ff80d7374a3843aa91" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41437 } } peers { permanent_uuid: "c44a3437a4fe44c5beb39ad2d0ccdd1f" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 45019 } } peers { permanent_uuid: "7d6da2a87e014aa3891c9a8d3b65a2ab" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 41239 } } }
I20260806 02:51:37.002023 25223 sys_catalog.cc:458] T 00000000000000000000000000000000 P c44a3437a4fe44c5beb39ad2d0ccdd1f [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:51:37.005848 25262 mvcc.cc:205] Tried to move back new op lower bound from 7315393318753517568 to 7315393318425362432. Current Snapshot: MvccSnapshot[applied={T|T < 7315393318753517568}]
I20260806 02:51:37.034066 25243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55c0f81eb98548ff80d7374a3843aa91 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "55c0f81eb98548ff80d7374a3843aa91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c0f81eb98548ff80d7374a3843aa91" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41437 } } peers { permanent_uuid: "c44a3437a4fe44c5beb39ad2d0ccdd1f" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 45019 } } peers { permanent_uuid: "7d6da2a87e014aa3891c9a8d3b65a2ab" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 41239 } } }
I20260806 02:51:37.034425 25243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55c0f81eb98548ff80d7374a3843aa91 [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:37.037650 25263 mvcc.cc:205] Tried to move back new op lower bound from 7315393318753517568 to 7315393318425362432. Current Snapshot: MvccSnapshot[applied={T|T < 7315393318753517568 or (T in {7315393318753517568})}]
I20260806 02:51:37.039112 25223 sys_catalog.cc:455] T 00000000000000000000000000000000 P c44a3437a4fe44c5beb39ad2d0ccdd1f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "55c0f81eb98548ff80d7374a3843aa91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c0f81eb98548ff80d7374a3843aa91" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41437 } } peers { permanent_uuid: "c44a3437a4fe44c5beb39ad2d0ccdd1f" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 45019 } } peers { permanent_uuid: "7d6da2a87e014aa3891c9a8d3b65a2ab" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 41239 } } }
I20260806 02:51:37.039386 25223 sys_catalog.cc:458] T 00000000000000000000000000000000 P c44a3437a4fe44c5beb39ad2d0ccdd1f [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:51:37.041742 25251 catalog_manager.cc:1347] Generated new cluster ID: a90847377236480f89c854c0b1b8def3
I20260806 02:51:37.041864 25251 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:51:37.049351 25209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d6da2a87e014aa3891c9a8d3b65a2ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55c0f81eb98548ff80d7374a3843aa91. Latest consensus state: current_term: 1 leader_uuid: "55c0f81eb98548ff80d7374a3843aa91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c0f81eb98548ff80d7374a3843aa91" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41437 } } peers { permanent_uuid: "c44a3437a4fe44c5beb39ad2d0ccdd1f" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 45019 } } peers { permanent_uuid: "7d6da2a87e014aa3891c9a8d3b65a2ab" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 41239 } } }
I20260806 02:51:37.049715 25209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d6da2a87e014aa3891c9a8d3b65a2ab [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:51:37.051254 25237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55c0f81eb98548ff80d7374a3843aa91 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "55c0f81eb98548ff80d7374a3843aa91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c0f81eb98548ff80d7374a3843aa91" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41437 } } peers { permanent_uuid: "c44a3437a4fe44c5beb39ad2d0ccdd1f" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 45019 } } peers { permanent_uuid: "7d6da2a87e014aa3891c9a8d3b65a2ab" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 41239 } } }
I20260806 02:51:37.051635 25237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55c0f81eb98548ff80d7374a3843aa91 [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:37.053483 25264 mvcc.cc:205] Tried to move back new op lower bound from 7315393318753517568 to 7315393318425362432. Current Snapshot: MvccSnapshot[applied={T|T < 7315393318753517568}]
I20260806 02:51:37.059278 25209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d6da2a87e014aa3891c9a8d3b65a2ab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "55c0f81eb98548ff80d7374a3843aa91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55c0f81eb98548ff80d7374a3843aa91" member_type: VOTER last_known_addr { host: "127.1.210.126" port: 41437 } } peers { permanent_uuid: "c44a3437a4fe44c5beb39ad2d0ccdd1f" member_type: VOTER last_known_addr { host: "127.1.210.125" port: 45019 } } peers { permanent_uuid: "7d6da2a87e014aa3891c9a8d3b65a2ab" member_type: VOTER last_known_addr { host: "127.1.210.124" port: 41239 } } }
I20260806 02:51:37.059603 25209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d6da2a87e014aa3891c9a8d3b65a2ab [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:51:37.103477 25251 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:51:37.105587 25251 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:51:37.134895 25251 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55c0f81eb98548ff80d7374a3843aa91: Generated new TSK 0
I20260806 02:51:37.135892 25251 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 02:51:37.257489 25207 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:37.237223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47846 (local address 127.1.210.126:41437)
0806 02:51:37.237676 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:37.237684 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:37.255462 (+ 17778us) server_negotiation.cc:407] Connection header received
0806 02:51:37.255804 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:37.255812 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:37.255905 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:37.255984 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:37.257210 (+ 1226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47846: BlockingRecv error: recv got EOF from 127.0.0.1:47846 (error 108)
Metrics: {"server-negotiator.queue_time_us":277}
W20260806 02:51:37.259831 25132 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:37.237203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36688 (local address 127.1.210.125:45019)
0806 02:51:37.237463 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:37.237470 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:37.258131 (+ 20661us) server_negotiation.cc:407] Connection header received
0806 02:51:37.258379 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:37.258385 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:37.258501 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:37.258575 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:37.259596 (+ 1021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36688: BlockingRecv error: recv got EOF from 127.0.0.1:36688 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260806 02:51:37.262408 25205 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:37.238175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43310 (local address 127.1.210.124:41239)
0806 02:51:37.238371 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:37.238379 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:37.260830 (+ 22451us) server_negotiation.cc:407] Connection header received
0806 02:51:37.261063 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:37.261069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:37.261159 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:37.261230 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:37.262176 (+ 946us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43310: BlockingRecv error: recv got EOF from 127.0.0.1:43310 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260806 02:51:37.270321 25132 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:37.268102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36702 (local address 127.1.210.125:45019)
0806 02:51:37.268260 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:37.268265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:37.268719 (+ 454us) server_negotiation.cc:407] Connection header received
0806 02:51:37.268905 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:37.268912 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:37.269045 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:37.269184 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:37.270021 (+ 837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36702: BlockingRecv error: recv got EOF from 127.0.0.1:36702 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260806 02:51:37.292904 25132 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:37.288986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36714 (local address 127.1.210.125:45019)
0806 02:51:37.289210 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:37.289217 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:37.291274 (+ 2057us) server_negotiation.cc:407] Connection header received
0806 02:51:37.291499 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:37.291505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:37.291620 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:37.291750 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:37.292654 (+ 904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36714: BlockingRecv error: recv got EOF from 127.0.0.1:36714 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260806 02:51:37.301672 25132 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:37.299507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36730 (local address 127.1.210.125:45019)
0806 02:51:37.299670 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:37.299676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:37.300154 (+ 478us) server_negotiation.cc:407] Connection header received
0806 02:51:37.300343 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:37.300350 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:37.300468 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:37.300583 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:37.301399 (+ 816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36730: BlockingRecv error: recv got EOF from 127.0.0.1:36730 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260806 02:51:37.312966 25132 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:37.309342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36736 (local address 127.1.210.125:45019)
0806 02:51:37.309574 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:37.309580 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:37.311291 (+ 1711us) server_negotiation.cc:407] Connection header received
0806 02:51:37.311564 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:37.311571 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:37.311688 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:37.311759 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:37.312738 (+ 979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36736: BlockingRecv error: recv got EOF from 127.0.0.1:36736 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
W20260806 02:51:37.328310 25132 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:37.325918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36748 (local address 127.1.210.125:45019)
0806 02:51:37.326131 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:37.326138 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:37.326720 (+ 582us) server_negotiation.cc:407] Connection header received
0806 02:51:37.326963 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:37.326970 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:37.327073 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:37.327195 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:37.328049 (+ 854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36748: BlockingRecv error: recv got EOF from 127.0.0.1:36748 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260806 02:51:37.352514 25132 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:37.349937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36752 (local address 127.1.210.125:45019)
0806 02:51:37.350136 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:37.350143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:37.350786 (+ 643us) server_negotiation.cc:407] Connection header received
0806 02:51:37.351034 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:37.351041 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:37.351149 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:37.351268 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:37.352224 (+ 956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36752: BlockingRecv error: recv got EOF from 127.0.0.1:36752 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260806 02:51:37.395069 25132 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:37.392538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36762 (local address 127.1.210.125:45019)
0806 02:51:37.392768 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:37.392777 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:37.393246 (+ 469us) server_negotiation.cc:407] Connection header received
0806 02:51:37.393474 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:37.393480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:37.393600 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:37.393752 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:37.394778 (+ 1026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36762: BlockingRecv error: recv got EOF from 127.0.0.1:36762 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260806 02:51:37.471012 25132 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:37.468983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36778 (local address 127.1.210.125:45019)
0806 02:51:37.469203 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:37.469210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:37.469549 (+ 339us) server_negotiation.cc:407] Connection header received
0806 02:51:37.469732 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:37.469738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:37.469833 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:37.469955 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:37.470792 (+ 837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36778: BlockingRecv error: recv got EOF from 127.0.0.1:36778 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260806 02:51:37.604971 25132 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:51:37.602805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36786 (local address 127.1.210.125:45019)
0806 02:51:37.603062 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:37.603069 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:37.603287 (+ 218us) server_negotiation.cc:407] Connection header received
0806 02:51:37.603469 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:37.603473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:37.603554 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:37.603650 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:37.604622 (+ 972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36786: BlockingRecv error: recv got EOF from 127.0.0.1:36786 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 02:51:37.619617 25220 catalog_manager.cc:1259] Loaded cluster ID: a90847377236480f89c854c0b1b8def3
I20260806 02:51:37.619717 25220 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7d6da2a87e014aa3891c9a8d3b65a2ab: loading cluster ID for follower catalog manager: success
I20260806 02:51:37.622503 25220 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7d6da2a87e014aa3891c9a8d3b65a2ab: acquiring CA information for follower catalog manager: success
I20260806 02:51:37.624456 25220 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7d6da2a87e014aa3891c9a8d3b65a2ab: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:51:37.717087 25234 catalog_manager.cc:1259] Loaded cluster ID: a90847377236480f89c854c0b1b8def3
I20260806 02:51:37.717206 25234 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c44a3437a4fe44c5beb39ad2d0ccdd1f: loading cluster ID for follower catalog manager: success
I20260806 02:51:37.719907 25234 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c44a3437a4fe44c5beb39ad2d0ccdd1f: acquiring CA information for follower catalog manager: success
I20260806 02:51:37.722548 25234 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c44a3437a4fe44c5beb39ad2d0ccdd1f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:51:37.871364 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:37.865044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36788 (local address 127.1.210.125:45019)
0806 02:51:37.865325 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:37.865336 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:37.865518 (+ 182us) server_negotiation.cc:407] Connection header received
0806 02:51:37.865687 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:37.865691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:37.865861 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:37.865979 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:51:37.866409 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:37.867197 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:37.868632 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:37.869138 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:37.869291 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:37.869953 (+ 662us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:51:37.870532 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:51:37.870658 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:37.870934 (+ 276us) server_negotiation.cc:299] Negotiation successful
0806 02:51:37.871013 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260806 02:51:37.884235 25334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:37.877994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43314 (local address 127.1.210.124:41239)
0806 02:51:37.878556 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:37.878560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:37.878583 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:51:37.878860 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:37.878864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:37.879054 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:37.879208 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:51:37.879633 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:37.880359 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:37.881760 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:37.882198 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:37.882300 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:37.882907 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:51:37.883516 (+ 609us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:51:37.883707 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:37.883775 (+ 68us) server_negotiation.cc:299] Negotiation successful
0806 02:51:37.883891 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":289,"threads_started":1}
I20260806 02:51:37.903713 25340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:37.890971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47856 (local address 127.1.210.126:41437)
0806 02:51:37.896782 (+ 5811us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:37.896787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:37.896804 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:51:37.896909 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:37.896912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:37.897098 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:37.897296 (+ 198us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:51:37.897810 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:37.898664 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:37.901075 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:37.901734 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:37.901854 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:37.902448 (+ 594us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:51:37.902979 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:51:37.903101 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:37.903260 (+ 159us) server_negotiation.cc:299] Negotiation successful
0806 02:51:37.903370 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5696,"thread_start_us":5590,"threads_started":1}
I20260806 02:51:37.904565 25334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:37.891105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43318 (local address 127.1.210.124:41239)
0806 02:51:37.891406 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:37.891416 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:37.898387 (+ 6971us) server_negotiation.cc:407] Connection header received
0806 02:51:37.898596 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:37.898602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:37.898744 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:37.898836 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:51:37.899827 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:37.900680 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:37.902407 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:37.903059 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:37.903180 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:37.903846 (+ 666us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:51:37.903984 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:51:37.904086 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:37.904220 (+ 134us) server_negotiation.cc:299] Negotiation successful
0806 02:51:37.904304 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260806 02:51:37.905686 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:37.890941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36802 (local address 127.1.210.125:45019)
0806 02:51:37.891174 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:37.891184 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:37.899566 (+ 8382us) server_negotiation.cc:407] Connection header received
0806 02:51:37.899771 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:37.899779 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:37.899876 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:37.899978 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:51:37.900585 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:37.901642 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:37.903505 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:37.904105 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:37.904193 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:37.904880 (+ 687us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:51:37.904993 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:51:37.905042 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:37.905326 (+ 284us) server_negotiation.cc:299] Negotiation successful
0806 02:51:37.905415 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 02:51:37.909889 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 24983
I20260806 02:51:37.912137 25046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:51:38.028438 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 24983
I20260806 02:51:38.053540 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 25058
I20260806 02:51:38.054813 25121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:51:38.167798 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 25058
I20260806 02:51:38.192519 1865 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu pid 25135
I20260806 02:51:38.193753 25197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:51:38.304611 1865 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5Fb5fs/build/asan/bin/kudu with pid 25135
2026-08-06T02:51:38Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4931 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10225 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (394105 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST